{"nbformat":4,"nbformat_minor":0,"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.1"},"colab":{"name":"2-CNN-for-sentence-classification (Kim 2014).ipynb","provenance":[],"collapsed_sections":[]}},"cells":[{"cell_type":"markdown","metadata":{"id":"LDjB0QH7wOnR"},"source":["# 2. CNN for Sentence Classification (Kim 2014)\n","- This is Keras implementation for **CNN for Sentence Classification** paper (Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.) \n","- There are four model variations\n","    - ```CNN-rand```: \"Our baseline model where all words are randomly initialized and then modified during training.\"\n","    - ```CNN-static```: \"A model with pre-trained vectors from word2vec. All words-including the unknown ones that are randomly initialized—are kept static and only the other parameters of the model are learned.\"\n","    - ```CNN-non-static```: \"Same as above but the pretrained vectors are fine-tuned for each task.\"\n","    - ```CNN-multichannel```: \"A model with two sets of word vectors. Each set of vectors is treated as a ‘channel’ and each filter is applied to both channels, but gradients are backpropagated only through one of the channels. Hence the model is able to fine-tune one set of vectors while keeping the other static. Both channels are initialized with word2vec.\"\n","    \n","</br>\n","<img src=\"http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/11/Screen-Shot-2015-11-06-at-8.03.47-AM.png\" style=\"width: 800px\"/>\n","    \n","- Dataset used is **Sentiment Analysis on Movie Reviews** from Kaggle\n","    - source: https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data"]},{"cell_type":"code","metadata":{"id":"y6bSyhhKwOnT","executionInfo":{"status":"ok","timestamp":1605374242426,"user_tz":420,"elapsed":2932,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","\n","from sklearn.model_selection import train_test_split\n","from multiprocessing import Pool\n","from gensim.models import Word2Vec\n","from tensorflow.keras.preprocessing.sequence import pad_sequences\n","from tensorflow.keras.preprocessing.text import Tokenizer\n","from tensorflow.keras.layers import *\n","from tensorflow.keras.models import *\n","from tensorflow.keras.utils import to_categorical\n","from tensorflow.keras.callbacks import *\n","from tensorflow.keras import optimizers"],"execution_count":1,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fVIve_jZwOnY"},"source":["## Dataset import and preprocessing"]},{"cell_type":"code","metadata":{"id":"84Q86RWKxotE","executionInfo":{"status":"ok","timestamp":1605374594895,"user_tz":420,"elapsed":67858,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"f842d170-5bee-4304-e1f7-8e8dfef242ec","colab":{"resources":{"http://localhost:8080/nbextensions/google.colab/files.js":{"data":"Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK","ok":true,"headers":[["content-type","application/javascript"]],"status":200,"status_text":""}},"base_uri":"https://localhost:8080/","height":109,"output_embedded_package_id":"1f83yKgTcVP0btpqXh9cJG03au3uA1td3"}},"source":["# if you are using Colab, first upload train.tsv\n","from google.colab import files\n","files.upload()"],"execution_count":3,"outputs":[{"output_type":"display_data","data":{"text/plain":"Output hidden; open in https://colab.research.google.com to view."},"metadata":{}}]},{"cell_type":"code","metadata":{"id":"95c10xLowOnY","executionInfo":{"status":"ok","timestamp":1605374594898,"user_tz":420,"elapsed":44445,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"43ab5f2a-1c66-4f1c-fda0-ef89dada4d84","colab":{"base_uri":"https://localhost:8080/"}},"source":["# exclude partial phrases, and import only sentences\n","df = pd.read_csv(\"train.tsv\", delimiter = '\\t').groupby(['SentenceId']).head(1)[['Phrase', 'Sentiment']]\n","df = df.reset_index(drop = True)\n","print(df.shape)\n","print(df.head())"],"execution_count":4,"outputs":[{"output_type":"stream","text":["(8529, 2)\n","                                              Phrase  Sentiment\n","0  A series of escapades demonstrating the adage ...          1\n","1  This quiet , introspective and entertaining in...          4\n","2  Even fans of Ismail Merchant 's work , I suspe...          1\n","3  A positively thrilling combination of ethnogra...          3\n","4  Aggressive self-glorification and a manipulati...          1\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"hJ_yxB6nwOnd","executionInfo":{"status":"ok","timestamp":1605374605903,"user_tz":420,"elapsed":550,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["num_tokens = 10000      # number of tokens\n","embed_dim = 50          # set word2vec embedding dim = 50"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"id":"MYDeeW8BwOng","executionInfo":{"status":"ok","timestamp":1605374606354,"user_tz":420,"elapsed":751,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["tokenizer = Tokenizer(num_words = num_tokens)\n","tokenizer.fit_on_texts(df['Phrase'])\n","sentences = tokenizer.texts_to_sequences(df['Phrase'])\n","word_idx = tokenizer.word_index"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"id":"_nXgv6K4wOni","executionInfo":{"status":"ok","timestamp":1605374606773,"user_tz":420,"elapsed":428,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["for i in range(len(sentences)):\n","    for j in range(len(sentences[i])):\n","        sentences[i][j] = str(sentences[i][j])"],"execution_count":7,"outputs":[]},{"cell_type":"code","metadata":{"id":"yfB4yKb_wOnl","executionInfo":{"status":"ok","timestamp":1605374614113,"user_tz":420,"elapsed":6834,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["# generate word2vec model with encoded sentences\n","model = Word2Vec(sentences = sentences, size = embed_dim, sg = 1, window = 3, min_count = 1, iter = 10, workers = Pool()._processes)  \n","model.init_sims(replace = True)"],"execution_count":8,"outputs":[]},{"cell_type":"code","metadata":{"id":"xzI5siTjwOno","executionInfo":{"status":"ok","timestamp":1605374614114,"user_tz":420,"elapsed":4178,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["sentences = pad_sequences(sentences, maxlen = len(max(sentences, key = len)), padding = 'post')"],"execution_count":9,"outputs":[]},{"cell_type":"code","metadata":{"id":"NAHrW7bjwOnt","executionInfo":{"status":"ok","timestamp":1605374614356,"user_tz":420,"elapsed":2263,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"2b60fb88-1d1a-4982-9a3f-a68a99c37d67","colab":{"base_uri":"https://localhost:8080/"}},"source":["embedding_matrix = np.zeros((len(word_idx) + 1, embed_dim))\n","\n","for word, idx in word_idx.items():\n","    try:\n","        embedding_matrix[idx] = model[str(idx)]\n","    except:\n","        pass\n","\n","print(len(word_idx))\n","print(embedding_matrix.shape)"],"execution_count":10,"outputs":[{"output_type":"stream","text":["15213\n","(15214, 50)\n"],"name":"stdout"},{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n","  \"\"\"\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"7GgLAZYVwOnv","executionInfo":{"status":"ok","timestamp":1605374616400,"user_tz":420,"elapsed":433,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"446ed741-6104-416b-d57e-7fb36b11258f","colab":{"base_uri":"https://localhost:8080/"}},"source":["# split data into train-test sets\n","X_train, X_test, y_train, y_test = train_test_split(sentences, df['Sentiment'])\n","y_train = to_categorical(y_train)\n","y_test = to_categorical(y_test)\n","\n","print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)"],"execution_count":11,"outputs":[{"output_type":"stream","text":["(6396, 47) (2133, 47) (6396, 5) (2133, 5)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"collapsed":true,"id":"o_TETteNwOny"},"source":["### CNN-rand\n","- All words are randomly initialized and the modified during training"]},{"cell_type":"code","metadata":{"id":"8_JLu0dhwOny","executionInfo":{"status":"ok","timestamp":1605374819706,"user_tz":420,"elapsed":541,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def cnn_rand(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n","    inputs = Input(shape = (max_len, ))\n","    embedded = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n","                         input_length = max_len, trainable = True)(inputs)\n","    conv_results = []\n","    for window in windows:\n","        x = Conv1D(filters, window, activation = 'relu')(embedded)\n","        x = Dropout(p)(x)\n","        x = MaxPooling1D(max_len - window + 1)(x)\n","        conv_results.append(x)\n","    conv_result = concatenate(conv_results)\n","    x = GlobalMaxPooling1D()(conv_result)\n","    x = Dense(50)(x)\n","    outputs = Dense(5, activation = 'softmax')(x)\n","    m = Model(inputs = inputs, outputs = outputs)\n","    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n","    return m"],"execution_count":24,"outputs":[]},{"cell_type":"code","metadata":{"id":"YfxNQooCwOn0","executionInfo":{"status":"ok","timestamp":1605374819970,"user_tz":420,"elapsed":564,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["cnn_rand = cnn_rand(embed_dim, num_tokens, X_train.shape[1])\n","callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"],"execution_count":25,"outputs":[]},{"cell_type":"code","metadata":{"id":"qHedZ5wYwOn4","executionInfo":{"status":"ok","timestamp":1605374859957,"user_tz":420,"elapsed":40057,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"8f5b6672-4abd-4ebe-8dc9-09d38e1dd691","colab":{"base_uri":"https://localhost:8080/"}},"source":["history = cnn_rand.fit(X_train, y_train, epochs = 10, callbacks = callbacks, validation_split = 0.1, batch_size = 256)"],"execution_count":26,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5744 - acc: 0.2646\n","Epoch 00001: val_acc improved from -inf to 0.29375, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 168ms/step - loss: 1.5744 - acc: 0.2646 - val_loss: 1.5679 - val_acc: 0.2937\n","Epoch 2/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5283 - acc: 0.3482\n","Epoch 00002: val_acc improved from 0.29375 to 0.30000, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 160ms/step - loss: 1.5283 - acc: 0.3482 - val_loss: 1.5535 - val_acc: 0.3000\n","Epoch 3/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4292 - acc: 0.4168\n","Epoch 00003: val_acc improved from 0.30000 to 0.37656, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 161ms/step - loss: 1.4292 - acc: 0.4168 - val_loss: 1.4815 - val_acc: 0.3766\n","Epoch 4/10\n","23/23 [==============================] - ETA: 0s - loss: 1.1899 - acc: 0.5257\n","Epoch 00004: val_acc improved from 0.37656 to 0.39687, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 162ms/step - loss: 1.1899 - acc: 0.5257 - val_loss: 1.3603 - val_acc: 0.3969\n","Epoch 5/10\n","23/23 [==============================] - ETA: 0s - loss: 0.9032 - acc: 0.6558\n","Epoch 00005: val_acc improved from 0.39687 to 0.40938, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 162ms/step - loss: 0.9032 - acc: 0.6558 - val_loss: 1.3088 - val_acc: 0.4094\n","Epoch 6/10\n","23/23 [==============================] - ETA: 0s - loss: 0.6277 - acc: 0.7832\n","Epoch 00006: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 162ms/step - loss: 0.6277 - acc: 0.7832 - val_loss: 1.3448 - val_acc: 0.3969\n","Epoch 7/10\n","23/23 [==============================] - ETA: 0s - loss: 0.4082 - acc: 0.8846\n","Epoch 00007: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 158ms/step - loss: 0.4082 - acc: 0.8846 - val_loss: 1.4134 - val_acc: 0.4062\n","Epoch 8/10\n","23/23 [==============================] - ETA: 0s - loss: 0.2456 - acc: 0.9359\n","Epoch 00008: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 158ms/step - loss: 0.2456 - acc: 0.9359 - val_loss: 1.5678 - val_acc: 0.3922\n","Epoch 9/10\n","23/23 [==============================] - ETA: 0s - loss: 0.1481 - acc: 0.9665\n","Epoch 00009: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 161ms/step - loss: 0.1481 - acc: 0.9665 - val_loss: 1.6911 - val_acc: 0.3938\n","Epoch 10/10\n","23/23 [==============================] - ETA: 0s - loss: 0.0916 - acc: 0.9826\n","Epoch 00010: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 163ms/step - loss: 0.0916 - acc: 0.9826 - val_loss: 1.8493 - val_acc: 0.3859\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"SV88XkpowOn6","executionInfo":{"status":"ok","timestamp":1605374666923,"user_tz":420,"elapsed":1196,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"fd763eea-5c48-445a-ee91-f531a0408503","colab":{"base_uri":"https://localhost:8080/","height":265}},"source":["plt.plot(history.history['acc'])\n","plt.plot(history.history['val_acc'])\n","plt.legend(['training', 'validation'], loc = 'upper left')\n","plt.show()"],"execution_count":15,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8deHLGRhy8aWBBIgkLAvIYAoRcEaXFBEBQQVBFFc66+1YlvrUltray31W4WyKruIqKiIW6GKbAmrEHYIEIIQEtbsyZzfHzNAEhJIyISbmfk8H495ZO4y934yj8w7Z+6cOUeMMSillHJ9dawuQCmllHNooCullJvQQFdKKTehga6UUm5CA10ppdyEt1UnDg0NNVFRUVadXimlXNKGDRtOGGPCyttmWaBHRUWRnJxs1emVUsolicjBirbpJRellHITGuhKKeUmrhjoIjJTRI6LyLYKtouIvC0ie0Vkq4h0d36ZSimlrqQy19DfA/4NzK5g+yAgxnHrBUx2/KyywsJC0tLSyMvLu5qHqzL8/PyIiIjAx8fH6lKUUtfAFQPdGPO9iERdZpc7gdnGPijMWhFpJCLNjDFHq1pMWloa9evXJyoqChGp6sNVCcYYMjMzSUtLIzo62upylFLXgDOuoYcDh0sspznWXUJExotIsogkZ2RkXLI9Ly+PkJAQDXMnEBFCQkL03Y5SHuSafihqjJlqjIk3xsSHhZXbjVLD3In0uVTKszijH/oRILLEcoRjnVJKKeBUTgGpmTkczMwm9UQON8U2plNEQ6efxxmBvhR4UkQWYv8w9PTVXD+vDU6dOsX8+fN5/PHHq/S4W2+9lfnz59OoUaMK9/njH/9Iv379GDhwYHXLVErVMsYYTpwrsAf2+eDOzOGQ4+fp3MJS+wfX87Um0EVkAdAfCBWRNOAlwMfxS0wBlgG3AnuBHGCM06u8Rk6dOsW77757SaAXFRXh7V3xU7Vs2bIrHvvVV1+tdn1KKevYbIZjZ/NIPZHDoazsUi3ug5nZZBcUX9i3jkBEUAAtQwIY3KU5LUMCiAoJJCo0gIigAPx8vGqkxsr0chlxhe0GeMJpFVlo4sSJ7Nu3j65du+Lj44Ofnx9BQUHs3LmT3bt3c9ddd3H48GHy8vJ45plnGD9+PHBxGINz584xaNAgrr/+elavXk14eDiffvop/v7+jB49mttvv5177rmHqKgoHnroIT777DMKCwv58MMPiY2NJSMjg/vvv5/09HT69OnDN998w4YNGwgNDbX4mVHKMxTbDOmncjmYmUNqZnapFvfBzBzyi2wX9vXxEiKD7UGdEB1MVEgALUMDiQoJJLyRP77e1/57m5aN5XIlr3y2nZT0M049ZvvmDXjpjg4Vbv/rX//Ktm3b2Lx5MytXruS2225j27ZtF7r9zZw5k+DgYHJzc+nZsydDhw4lJCSk1DH27NnDggULmDZtGvfddx8fffQRo0aNuuRcoaGhbNy4kXfffZc333yT6dOn88orr3DTTTfxwgsvsHz5cmbMmOHU318pBYXFNtJO5toD+0SJwM7K4XBWDoXFF6flrOtdh5YhAbQMCeQXbcNoGWIP7JYhATRv5I9XndrV8aDWBnptkJCQUKoP99tvv83HH38MwOHDh9mzZ88lgR4dHU3Xrl0B6NGjB6mpqeUe++67776wz5IlSwBYtWrVheMnJiYSFBTk1N9HKU+TX1TM1rTTJKVmsSH1JHuOn+PIqVyKbRdDO9DXi5YhgcQ2rc8tHZoSFRJAi2D75ZEm9f2oU8tC+3JqbaBfriV9rQQGBl64v3LlSr799lvWrFlDQEAA/fv3L7ePd926dS/c9/LyIjc3t9xjn9/Py8uLoqIiJ1eulGc6nVvIxoMnSUrNIik1iy1ppylwXCZpHRZIl8hG3Nm1uaOlbW95h9bzdZsuvrU20K1Qv359zp49W+6206dPExQUREBAADt37mTt2rVOP3/fvn1ZtGgRzz//PF9//TUnT550+jmUcidHT+eSlHqSpAP2AN917CzGgHcdoWN4Qx7q05KeUcH0aBlESL26Vz6gi9NALyEkJIS+ffvSsWNH/P39adKkyYVtiYmJTJkyhbi4ONq1a0fv3r2dfv6XXnqJESNGMGfOHPr06UPTpk2pX7++08+jlCuy2Qz7Ms6xPjWL5FR7KzztpP0dcKCvF91bBjGoYzN6RgfRNbIRAb6eF29i76Ry7cXHx5uyE1zs2LGDuLg4S+qpDfLz8/Hy8sLb25s1a9YwYcIENm/eXK1jevpzqlxXQZGNbemnHa3vk2w4mMXJHHt/7tB6vvSMCr5wi2tWH28vzxgNXEQ2GGPiy9vmef/CarFDhw5x3333YbPZ8PX1Zdq0aVaXpNQ1czavkI2HTpGcmsX6A1lsPnzqQjfB6NBABsY1oWe0PcCjQgLc5rq3M2mg1yIxMTFs2rTJ6jKUuiaOn8mzX/92fIC54+gZbMb+pZwOzRsysldLekYFER8VTFh997/+7Qwa6EqpGmeMYf+JbEfr+yTJB7M4mJkDgJ9PHbpFBvHkTTH0jAqiW4sg6tXVaLoa+qwppWrMjqNnmPXjAb7bcZzM7AIAggN9iW8ZxKheLYmPCqJjeEN8POT6d03TQFdKOZXNZli5+zgzVh3gx72Z+Pt4cUuHJvRqFULPqGBahwXq9e8aooGulHKKnIIiPtp4hFk/HmB/RjZNG/jxfGIsIxIiaRTga3V5HkHf51RDvXr1AEhPT+eee+4pd5/+/ftTtntmWZMmTSInJ+fC8q233sqpU6ecV6hSNejn03m8sXwnfV7/Ly9+so36db351/Cu/PD8jUzo31rD/BrSFroTNG/enMWLF1/14ydNmsSoUaMICAgAKjccr1JW25p2ihmrDvDF1qPYjOGWDk0Ze300PVoG6SUVi2gLvYSJEyfyzjvvXFh++eWXee211xgwYADdu3enU6dOfPrpp5c8LjU1lY4dOwKQm5vL8OHDiYuLY8iQIaXGcpkwYQLx8fF06NCBl156CbAP+JWens6NN97IjTfeCNiH4z1x4gQAb731Fh07dqRjx45MmjTpwvni4uJ45JFH6NChA7/85S8rHDNGKWcqthmWbzvKvVNWM/jfP/LdjuM82CeK/z13I5NH9SA+KljD3EK1t4X+5UT4+SfnHrNpJxj01wo3Dxs2jF/96lc88YR9ePdFixbx1Vdf8fTTT9OgQQNOnDhB7969GTx4cIV/tJMnTyYgIIAdO3awdetWunfvfmHbn//8Z4KDgykuLmbAgAFs3bqVp59+mrfeeosVK1ZcMu75hg0bmDVrFuvWrcMYQ69evfjFL35BUFBQpYfpVcoZzuUXsSjpMLNWH+BwVi4RQf784bY4hvWMpL6fj9XlKYfaG+gW6NatG8ePHyc9PZ2MjAyCgoJo2rQpzz77LN9//z116tThyJEjHDt2jKZNm5Z7jO+//56nn34agM6dO9O5c+cL2xYtWsTUqVMpKiri6NGjpKSklNpe1qpVqxgyZMiFUR/vvvtufvjhBwYPHlzpYXqVqo7DWTm8vzqVD5IOcza/iPiWQfxuUBw3t2/iMV+1dyW1N9Av05KuSffeey+LFy/m559/ZtiwYcybN4+MjAw2bNiAj48PUVFR5Q6beyUHDhzgzTffJCkpiaCgIEaPHn1VxzmvssP0KlVVxhg2HjrJjFUHWL7tZ0SE2zo1Y+z10XSJrHjeXGW9Sv2LFZFEEdklIntFZGI521uKyHcislVEVopIhPNLvTaGDRvGwoULWbx4Mffeey+nT5+mcePG+Pj4sGLFCg4ePHjZx/fr14/58+cDsG3bNrZu3QrAmTNnCAwMpGHDhhw7dowvv/zywmMqGrb3hhtu4JNPPiEnJ4fs7Gw+/vhjbrjhBif+tkpdVFhsY+mWdO56dzVDJ69h1Z4TjO/XmlXP38jbI7ppmLuAykwS7QW8A9wMpAFJIrLUGJNSYrc3gdnGmPdF5CbgdeCBmii4pnXo0IGzZ88SHh5Os2bNGDlyJHfccQedOnUiPj6e2NjYyz5+woQJjBkzhri4OOLi4ujRowcAXbp0oVu3bsTGxhIZGUnfvn0vPGb8+PEkJibSvHlzVqxYcWF99+7dGT16NAkJCQCMGzeObt266eUV5VSncwpZkHSI91encvR0HtGhgfzpzg4M7RHhkUPQurIrDp8rIn2Al40xtziWXwAwxrxeYp/tQKIx5rDYPy08bYxpcLnj6vC514Y+p6oiB05kM+vHAyzekEZOQTF9WoUw9vpobopt7FLTrnma6g6fGw4cLrGcBvQqs88W4G7gX8AQoL6IhBhjMssUMh4YD9CiRYvKVa+UchpjDGv2ZzJz1QG+23kc7zrC4C7hPHx9FB2aN7S6PFVNzno/9Rvg3yIyGvgeOAIUl93JGDMVmAr2FrqTzq2UuoL8omI+23KUmasOkHL0DMGBvjx1YxtG9WlJ4/p+VpennKQygX4EiCyxHOFYd4ExJh17Cx0RqQcMNcZc1XfXjTH6xQQnsWo2KlV7ZGUXMG/tQWavPUjG2XxiGtfjr3d34q5u4fj5eFldnnKyygR6EhAjItHYg3w4cH/JHUQkFMgyxtiAF4CZV1OMn58fmZmZhISEaKhXkzGGzMxM/Py09eWJjDG8u3Ifb3+3h/wiG/3ahvHmvdH0iwnV15Ybu2KgG2OKRORJ4CvAC5hpjNkuIq8CycaYpUB/4HURMdgvuTxxNcVERESQlpZGRkbG1TxcleHn50dEhMv2IFVXKaegiN98uIVlP/3MrZ2a8uzAtsQ00cnGPUGtmiRaKVU9h7NyeGR2MruPneWFQXGMuyFaW+RuRieJVsoDrNmXyePzNlBkM8wak8Av2oZZXZK6xjTQlXJxxhjmrj3IK5+l0DIkgGkPxtMqrJ7VZSkLaKAr5cIKimy8tHQbC9YfZkBsYyYN76qjH3owDXSlXFTG2XwmzN1A8sGTPHFja/7fze3w0m94ejQNdKVc0E9ppxk/J5mTOQX834hu3NGludUlqVpAA10pF/Pp5iP8dvFWQuvVZfFj19ExXL+yr+w00JVyEcU2w9+/2sWU/+0jISqYd0d1J7Re3Ss/UHkMDXSlXMCZvEKeWbCJFbsyGNmrBS/d0QFfb50xSJWmga5ULbcv4xyPzE7mUGYOr93VkVG9W1pdkqqlNNCVqsVW7DrO0ws24etVh3njetGrVYjVJalaTANdqVrIGMN/vt/PG8t3Ete0AVMf7EFEUIDVZalaTgNdqVomr7CY5z/ayqeb07mtczP+fk9nnQpOVYr+lShVi6SfymX8nGS2p5/huVva8Xj/1jq4lqo0DXSlaonk1Cwem7uBvEIb0x+MZ0BcE6tLUi5GA12pWmDh+kO8+Ok2IoICWDi+B20a6/jlquo00JWyUGGxjT99nsLsNQfp1zaM/xvejYYBOriWujoa6EpZJPNcPk/M38ja/Vk82q8Vv02M1cG1VLVooCtlgZT0MzwyO5mMc/n8c1gXhnTTqQJV9VXqu8Mikigiu0Rkr4hMLGd7CxFZISKbRGSriNzq/FKVcg/LfjrK0MmrKbYZPny0j4a5cporttBFxAt4B7gZSAOSRGSpMSalxG5/ABYZYyaLSHtgGRBVA/Uq5bJsNsOkb3fz9n/30r1FI6aM6kHjBn5Wl6XcSGUuuSQAe40x+wFEZCFwJ1Ay0A3QwHG/IZDuzCKVcnVn8wp59oMtfLvjGPfFR/CnuzpS19vL6rKUm6lMoIcDh0sspwG9yuzzMvC1iDwFBAIDyzuQiIwHxgO0aNGiqrUq5ZIOZmYz7v1k9p/I5uU72vPQdVH6ZSFVI5w1/uYI4D1jTARwKzBHRC45tjFmqjEm3hgTHxamM5Ir9/fDngwG//tHMs7lM+fhBEb3jdYwVzWmMi30I0BkieUIx7qSxgKJAMaYNSLiB4QCx51RpFKuxhjDjFUH+MuyHcQ0rs+0B+NpEaKDa6maVZkWehIQIyLRIuILDAeWltnnEDAAQETiAD8gw5mFKuUq8ouK+c2HW3ntix3c3L4JSx6/TsNcXRNXbKEbY4pE5EngK8ALmGmM2S4irwLJxpilwK+BaSLyLPYPSEcbY0xNFq5UbVRUbOOp+Zv4OuUYvxoYw9M3xVBHvyykrpFKfbHIGLMMe1fEkuv+WOJ+CtDXuaUp5VqMMfz+4218nXKMl+5oz5i+0VaXpDyMTkqolJO8sXwXHyQf5umb2miYK0tooCvlBFO/38eU/+1jZK8WPHtzW6vLUR5KA12pavow+TB/WbaT2zo349U7O2q3RGUZDXSlquGblGNMXPITN8SE8s/7uupoicpSGuhKXaW1+zN5Yv5GOoY3ZMqoHvh668tJWUv/ApW6CtuOnOaR95NpERzArNE9CayrI1Er62mgK1VFB05kM3rWeur7eTP74QSCA32tLkkpQANdqSo5diaPB2asw2Zg9theNG/kb3VJSl2gga5UJZ3OKeTBGes5mV3Ae2N60qZxPatLUqoUvfCnVCXkFhTz8PtJHDiRzawxPekc0cjqkpS6hLbQlbqCwmIbE+ZtYNOhk/xreFf6tgm1uiSlyqUtdKUuw2YzPPfhFlbuyuD1uzsxqFMzq0tSqkLaQleqAsYYXv08hU82p/PcLe0YkaCzbKnaTQNdqQr8+797eW91KmOvj+bx/q2tLkepK9JAV6occ9ce5B/f7ObubuH8/tY4HZ9FuQQNdKXK+HxrOi9+uo0BsY15457OOkGFchka6EqV8MOeDJ79YDPxLYN4Z2R3fLz0JaJch/61KuWw6dBJHp2zgdZh9Zj+UE/8fLysLkmpKqlUoItIoojsEpG9IjKxnO3/FJHNjttuETnl/FKVqjl7j59lzHtJhNary+yHE2jo72N1SUpV2RX7oYuIF/AOcDOQBiSJyFLHPKIAGGOeLbH/U0C3GqhVqRpx5FQuD8xYj3edOswZm0DjBn5Wl6TUValMCz0B2GuM2W+MKQAWAndeZv8RwAJnFKdUTcs8l88DM9ZxLr+I2Q8n0DIk0OqSlLpqlQn0cOBwieU0x7pLiEhLIBr4bwXbx4tIsogkZ2RkVLVWpZzqXH4RY95L4sjJXGY81JP2zRtYXZJS1eLsD0WHA4uNMcXlbTTGTDXGxBtj4sPCwpx8aqUqL7+omEfnJLM9/QzvjuxOQnSw1SUpVW2VCfQjQGSJ5QjHuvIMRy+3qFqu2Gb41cLN/Lg3k78N7cyAuCZWl6SUU1Qm0JOAGBGJFhFf7KG9tOxOIhILBAFrnFuiUs5jjOEPn2zjy20/84fb4hjaI8LqkpRymisGujGmCHgS+ArYASwyxmwXkVdFZHCJXYcDC40xpmZKVar63vx6FwvWH+Lx/q0Zd0Mrq8tRyqkqNXyuMWYZsKzMuj+WWX7ZeWUp5XwzVh3gnRX7GJEQyXO3tLO6HKWcTr8pqjzCko1p/OnzFAZ1bMprd3XSwbaUW9JAV27vux3HeG7xVq5rHcKk4V3x0sG2lJvSQFdubf2BLB6ft5EOzRsw9cF46nrr+CzKfWmgK7eVkn6Gse8nER7kz6zRPalXV2dcVO5NA125pUOZOTw0az316nozZ2wvQurVtbokpWqcBrpyO8fP5jFqxjoKi23MGZtAeCN/q0tS6prQQFdu5XRuIQ/OWM+Jc/nMGt2TNo3rW12SUteMBrpyG7kFxTzyfjL7Ms4xZVQPurUIsrokpa4p/ZRIuYWCIhuPzd1A0sEs3h7ejX5tdfA35Xm0ha5cXlGxjWcWbuJ/uzN4fUgn7ujS3OqSlLKEBrpyaTabYeKSny4MtjU8oYXVJSllGQ105bKMMbz6eQqLN6Txq4ExOtiW8nga6Mpl/ePr3by3OpVx10fzzIAYq8tRynIa6MolTV65j3+v2MuIhEh+f1ucDralFBroygXNWZPKG8t3MrhLcx05UakSNNCVS/loQxovfrqdgXGN+cd9XXTkRKVK0EBXLmP5tqM8t3gL17UO4d/3d8fHS/98lSpJXxHKJfxvdwZPLdhEl8hGTHswHj8fHQZXqbIqFegikigiu0Rkr4hMrGCf+0QkRUS2i8h855apPFlSahaPzkkmpnF93hudQKAOg6tUua74yhARL+Ad4GYgDUgSkaXGmJQS+8QALwB9jTEnRaRxTRWsPMtPaad5eFYSzRv5M3tsAg0DfKwuSalaqzIt9ARgrzFmvzGmAFgI3Flmn0eAd4wxJwGMMcedW6byRLuPneXBmeto4O/DvHG9CNUxzZW6rMoEejhwuMRymmNdSW2BtiLyo4isFZHE8g4kIuNFJFlEkjMyMq6uYuURDmZmM2r6Ony86jD/kV40a6hjmit1Jc76UNQbiAH6AyOAaSLSqOxOxpipxph4Y0x8WJiOhqfKd/R0LiOnr6Og2Mbccb1oGRJodUlKuYTKBPoRILLEcoRjXUlpwFJjTKEx5gCwG3vAK1UlJ87lM2r6Ok7lFDL74QTaNtEJKpSqrMoEehIQIyLRIuILDAeWltnnE+ytc0QkFPslmP1OrFN5gPOzDR05lcvM0T3pHHHJmzyl1GVcMdCNMUXAk8BXwA5gkTFmu4i8KiKDHbt9BWSKSAqwAnjOGJNZU0Ur95OdX8SYWevZc/ws/3kgnoToYKtLUsrliDHGkhPHx8eb5ORkS86tape8wmLGvp/Emn2ZvDuyO4kdm1ldklK1lohsMMbEl7dNv6GhLFVYbOPJ+Zv4cW8m/7i3i4a5UtWgX/1Xlim2GX7z4Ra+3XGMV+/swNAeEVaXpJRL00BXljDG8IdPtvHp5nR+m9iOB/tEWV2SUi5PA11dc8YY/rJsBwvWH+KJG1vzeP82VpeklFvQQFfX3Nvf7WXaDwd4qE9LfvPLdlaXo5Tb0EBX19T0H/bzz293M7R7BC/d0UFnG1LKiTTQ1TWzcP0hXvtiB4M6NuWNoZ2oo7MNKeVUGujqmvhsSzovfPwTv2gbxr+Gd8NbZxtSyun0VaVq3Hc7jvHsB5vpGRXMlFE98PXWPzulaoK+slSNWr3vBBPmbaR98wbMeCgef1+dOk6pmqKBrmrMxkMnGfd+MlEhAbw/JoH6fjrbkFI1SQNd1YiU9DOMnrmesPp1mTu2F0GBvlaXpJTb00BXTrcv4xwPzlxHYF1v5o7tReMGflaXpJRH0EBXTpV2ModR09cBMHdcLyKDAyyuSCnPoYGunOb4mTxGTl9Hdn4Rsx/uReuwelaXpJRH0eFzlVOczC7ggRnryTibz9xxvWjfvIHVJSnlcTTQVbVlnstn9KwkDmRm897onnRvEWR1SUp5JA10VS37M84xelYSx87kMWVUd65rE2p1SUp5rEpdQxeRRBHZJSJ7RWRiOdtHi0iGiGx23MY5v1RV2ySlZnH35NVk5xexcHxvboptYnVJSnm0K7bQRcQLeAe4GUgDkkRkqTEmpcyuHxhjnqyBGlUt9NmWdH69aAsRwf68NzqBFiHam0Upq1WmhZ4A7DXG7DfGFAALgTtrtixVWxljmLxyH08t2ETXyEYsmXCdhrlStURlAj0cOFxiOc2xrqyhIrJVRBaLSGR5BxKR8SKSLCLJGRkZV1GuslJRsY3ff7KNN5bvZHCX5swZl0CjAP0GqFK1hbP6oX8GRBljOgPfAO+Xt5MxZqoxJt4YEx8WFuakU6tr4Vx+EeNmJzN/3SEe79+aScO6UtdbB9pSqjapTC+XI0DJFneEY90FxpjMEovTgb9VvzRVWxw7k8eYWUnsOnaW1+/uxIiEFlaXpJQqR2UCPQmIEZFo7EE+HLi/5A4i0swYc9SxOBjY4dQqlWV2/nyGMbOSOJNbyIyH4unfrrHVJSmlKnDFQDfGFInIk8BXgBcw0xizXUReBZKNMUuBp0VkMFAEZAGja7BmdY38sCeDCXM3EljXi0WP9aFD84ZWl6SUugwxxlhy4vj4eJOcnGzJudWVLUo+zO+W/ESbxvWYNaYnzRr6W12SUgoQkQ3GmPjytuk3RVUpxhj++c1u3v7vXm6ICeXdkd11YgqlXIQGurqgoMjG8x9t5eNNRxgWH8lrQzrio5M5K+UyNNAVAKdzCnl0bjJr92fxm1+25Ykb2yAiVpellKoCDXTF4awcxryXxMHMbCYN68pd3cr73phSqrbTQPdwW9NO8fB7yRQUFTNnbC96twqxuiSl1FXSQPdg36Yc46kFmwip58vC8b1o07i+1SUppapBA91Dvb86lVc+206n8IZMf6gnYfXrWl2SUqqaNNA9jM1m+MuyHUxfdYCb2zfhX8O7EuCrfwZKuQN9JXuQvMJinv1gM19u+5nR10Xx4u3t8aqjPVmUchca6B4i81w+42Yns/nwKV68vT1jr4+2uiSllJNpoHuAkvN+Th7Zg8SOTa0uSSlVAzTQ3VxSahaPzE7GS4SF43vTrUWQ1SUppWqIBrob+2xLOr/+cAsRQTrvp1KeQAPdDRljmPK//byxfCcJUcFMfbCHThWnlAfQQHczRcU2/rh0O/PXHeKOLs35+z2d8fPRqeKU8gQa6G7kXH4RT87fyMpdGTzevzW/+WU76mi3RKU8hga6m9B5P5VSGuhuQOf9VEoBVGr2AhFJFJFdIrJXRCZeZr+hImJEpNzpkZTzrdh1nHsnr8FmDIse66NhrpQHu2ILXUS8gHeAm4E0IElElhpjUsrsVx94BlhXE4Wq0vYcO8sby3fy7Y7jxDatz8zRPWneSOf9VMqTVeaSSwKw1xizH0BEFgJ3Aill9vsT8AbwnFMrVKUcP5PHP7/dzQdJhwn09eb5xFjG9I3SnixWKS6ColwoLHEryoWiAvDyBq+64OUL3r72n2VvdXSKP+U8lQn0cOBwieU0oFfJHUSkOxBpjPlCRCoMdBEZD4wHaNFCP7Sriuz8Iv7z/X6mfb+fIpuNh66L4qmbYggOtKB/uc0Gpw5Cxi44scvxcw/YCkuHlXdd8PJxLDvul7eu1L4VrfMt8fiSoVhiXR1vELHXV5R3MVwLc6EwBwrz7D+LHD9LhXBe6X1KPba8xznu2wqr91zW8b405MsL/6td5x8EzbtBUJT9uVFurdofiopIHeAtYPSV9jXGTAWmAsTHx5vqntsTFBXbWJh0mEnf7uHEuXxu69yM397SjiCbd9oAAA7KSURBVJYhgTV/8uJCyNpvD+wL4b3THt5FeRf3q9cEQtuCTyMoyrc/rvA0FDvun19XXFB6nSl2csFiD8irDdk6PuATAD7+4ONnv+/t+BkQ7FhfYl2pffxLr/PysbfeiwtK34oKSj8PxQUlnp/LrCvIgeJTJdaVPFaJdRUJCIHwHhAe7/jZ3f47KbdSmUA/AkSWWI5wrDuvPtARWOmYVLgpsFREBhtjkp1VqKcxxvBNyjH+unwn+zOySYgKZtqDPWpmLJbCXMjcezG4M3baf2btA1vRxf0atoCwthD9CwhrB6Ht7Mv+V1mTrbh0aBXlXxpQ5YVWqX3LhKWt0N6i9/EvcSsZuo6bt3/pEPb2t18icWXGlPjH6bidPQpHNjpuybDnG8DRlgpu5Qh3R9A37WR/LpTLEmMu31AWEW9gNzAAe5AnAfcbY7ZXsP9K4DdXCvP4+HiTnKx5X55Nh07y+rKdrE/NonVYIBMHxTEwrjFS3bfM+WchY/fFlvb5AD+ZyoUXudSBoGgIi7WHdVisvfUd2hbq1qvur6aslncGjm6GtGQ4ssF+O3vUvq2ONzTpCBHxF0M+pI1e569lRGSDMabcnoRXbJIYY4pE5EngK8ALmGmM2S4irwLJxpilzi3Xcx3MzOZvX+3ii61HCa1Xlz8P6ciw+Ei8var4gsrJKt3SPn+d+0yJN1Z1fCA0Bpp1gc7DLoZ3cGttpbkzvwYQ3c9+O+9Muj3Yz4f8loWQNN2+rW4D+zX4CyHfA+rr8Mu11RVb6DVFW+gXZWUX8H//3cPctQfxrlOH8f1a8Ui/VtSrW4lLABm7Yf/K0uGdnXFxu7d/6ZZ2WKz9cklQtOtfYlA1w1YMJ3ZfbMGnJcOx7Rc/82gQYb8GH97DHvTNuta+d2/GQME5yDtd+ubla//swD/Y/tO3nst9WFytFrqqOXmFxcz88QCTV+wju6CIYT1b8OzAGBo3qGQLOeVT+OgR+7Xlug3twd32Fkd4t7MHd8NIfcusqqaOFzSOs9+6jbKvK8yFo1vt1+HPB/0Ox5tzqQNhcaVDPiyueg2GigL5ktupircZ25XP4+V7Mdz9gyEgyP4Bcql1waXX+TWqta8pbaFboNhm+HjTEf7x9S6Ons5jYFxjnk+MJaZJ/codwBhY+y589XuI6AlDp0Gjli7X0lAuLvuE48PWDReDPvekfZtPgL3lfj7k6zWB/DPODWSfQPBrWIVbA/sH57lZ9suSuVmQk+m4f/LSdRX2whJ7R4CSgX8h+MuuC7l439s5XYwv10LXQL/Gvt+dwetf7mTH0TN0iWjIC7fG0btVSOUPYCuG5S/A+v9A3GC4e6q9t4ZSVjMGTh6AtA0XW/FHt9jfQZanyoF8/tbIHs5ePjX7u+SfKRHyWWXuZ5b5x3DSvq4ot+Jj+ta7GPY3/BraD76q0vSSSy2Qkn6G17/cwQ97ThAZ7M//jejGbZ2aVW1424IcWPII7Pwc+jwJN/+p1r71Ux5IxN4VMrgVdL7Xvq6oAI5vh9xT1zaQq0vkYr1UYUL1wtwKgv9k6XXeNdPxQAO9hqWfyuUfX+9myaY0Gvr78OLt7RnVuwV1vav4Vf1zGbBguL3Vk/gG9H6sZgpWypm8fe29ZDyFjz80DLffLKCBXkPO5BUyeeU+Zq46gAHG92vF4/3b0ND/KlolJ/bCvKFw9mcYNgfi7nB6vUop16eB7mQFRTbmrTvI29/t4VRuIUO6hvP/ftmWiKCrnKD50Fp7y1zqwEOfQ2RP5xaslHIbGuhOYozhi5+O8rfluziUlcP1bUKZOCiWjuENr/6g2z+BJeOhYQSMWmy/NqmUUhXQQHeC9Qey+POyHWw5fIrYpvV5/+EE+sWEXv1X9Y2BNe/A13+AyAQYvgACq9ATRinlkTTQq2Hv8XO8sXwn36Qco2kDP/5+T2fu7h6BV3UmZrYVw/KJsH6qdktUSlWJBvpVyM4v4s2vdzF7zUH8fbx47pZ2PNw3Gn/fak4yUZADH42DXV9ot0SlVJVpoFfRD3symPjRT6SfzmVkrxY8O7AtIfXqVv/A5zJgwTD7N+8G/Q16PVr9YyqlPIoGeiWdzinktS9S+HBDGq3CAvnw0T7ERzlpgoATe2DePXD2GAyfB7G3Oee4SimPooFeCcu3/cyLn24jK7uAJ25szVM3xThvDs+Da2DhCBAvGP25fWAjpZS6Cq4X6OmbYOcy6Dm2xsdlPn42j5eXbmfZTz/TvlkDZo3uWb1uiGVt/xiWPKrdEpVSTuF6gZ66Cr7/O6z6J3QcCr0nQPOuTj2FMYYlG4/w6ucp5BYW89wt7RjfrxU+VZ1oouITwJp/O7ol9oYRC3R+R6VUtbleoF/3FLS71d6tb9Nc2LoQWlxnD/bY2+xjOVdD2skcfvfxNr7fnUF8yyD+OrQzbRo7cfD+kt0S298FQ/6jMwQppZzCtYfPzTsNm+bBuilw6iA0agEJ46HbA+DfqEqHstkMc9cd5I0vd2KA5xNjeaB3y6qNhnglBdmObonL7P+YBr6q3RKVUlVS7fHQRSQR+Bf2OUWnG2P+Wmb7Y8ATQDFwDhhvjEm53DGdOh66rRh2fQlrJ8PBVfZxlruNhF6PQUjrKz58X8Y5Jn60laTUk/RrG8ZfhnS8+rFXKnLuOMwfZp+gN/EN6DXeucdXSnmEagW6iHgBu4GbgTQgCRhRMrBFpIEx5ozj/mDgcWNM4uWOW2MTXBzdAmunwLbFUFxon5Kt9wSI/sUlM/oUFduY+sN+Jn27B38fL168vT1Du4df/Vf2K3JiD8wdag/1e2ZC7K3OPb5SymNUd4KLBGCvMWa/42ALgTuBC4F+PswdAgFrruOAfRb7IZNh4MuQPBOSZ8DsO6Fxe3uwd7oXfPzZnn6a5z/ayrYjZxjUsSmv3NmBxvVr4Fr2+W6Jdbxh9BcQ0cP551BKKSoX6OHA4RLLaUCvsjuJyBPA/wN8gZvKO5CIjAfGA7Ro0aKqtVZN/SZw4wtw/bOw7SP75ZilT2G+fZk1QYP59YF4CgOaMnlkdwZ1alYzNWxbAh8/Bo0iYeRiCK7CzCdKKVVFTvtEzhjzjjGmNfA88IcK9plqjIk3xsSHhYU569SX5+Nnv57+2A/sSlzA6oI29E57jx/qPsPqtgsYFHzU+ec0Bn58GxaPsU+SO/YbDXOlVI2rTAv9CBBZYjnCsa4iC4HJ1SnK2bLzi/j7V7t4f42hecOJvDWoAb2OL7Z3e0xZbO8L3nsCxN4OXtXsyWkrhi+fh6Rp0GEI3DVFuyUqpa6JyqRXEhAjItHYg3w4cH/JHUQkxhizx7F4G7CHWuL73Rm8sMQ+mNaDvVvyXGIs9ep6A/Fw4+/sob5uCnz4EDSMtHd77P5glbs9AvZuiYvHwu4v4bqnYeAr2i1RKXXNVLbb4q3AJOzdFmcaY/4sIq8CycaYpSLyL2AgUAicBJ40xmy/3DFrrJeLw+mcQv70RQqLHYNpvTG0Mz0rGkzrfLfHdVMg9Qd7t8eu99u7PYa2qdwJzx2H+ffZe9kM+hskPOK8X0YppRyq3Q+9JtRkoJccTOvRfq14ekAVBtM6utUe7D99CMUFEOPo9tiq/yXdHi/I2G2fxPlchnZLVErVKI8J9LKDaf3tns5XP5jWueP2bo9J0yE7A8Li7MHe+b7SMwgdXA0LRoCXD9z/AYRrt0SlVM1x+0AvO5jWMwNinDeYVlG+o9vju/DzT+AfDPFjoOc4OLTG0S2xpX20xKCo6p9PKaUuw60DveRgWj1aBvGGswfTOs8YOPijvT/7zi/sg4DZiuwDgw2fp6MlKqWuiep+U7RWKjuY1iuDOzh/MK2SRCDqevst6wCsnwa2Qvu8n9otUSlVC7hkoJccTOuGmFD+MqQTkcFOHkzrcoKjIfEv1+58SilVCS4X6IuSDvOHT7fh512Hv9/TmXt6RDh/MC2llHJBLhfo0WGBDIhtXHODaSmllItyuUDvGRVc8ReElFLKg+n30pVSyk1ooCullJvQQFdKKTehga6UUm5CA10ppdyEBrpSSrkJDXSllHITGuhKKeUmLBttUUQygINX+fBQ4IQTy3F1+nyUps/HRfpclOYOz0dLY0xYeRssC/TqEJHkioaP9ET6fJSmz8dF+lyU5u7Ph15yUUopN6GBrpRSbsJVA32q1QXUMvp8lKbPx0X6XJTm1s+HS15DV0opdSlXbaErpZQqQwNdKaXchMsFuogkisguEdkrIhOtrscqIhIpIitEJEVEtovIM1bXVBuIiJeIbBKRz62uxWoi0khEFovIThHZISJ9rK7JKiLyrON1sk1EFoiIW0535lKBLiJewDvAIKA9MEJE2ltblWWKgF8bY9oDvYEnPPi5KOkZYIfVRdQS/wKWG2NigS546PMiIuHA00C8MaYj4AUMt7aqmuFSgQ4kAHuNMfuNMQXAQuBOi2uyhDHmqDFmo+P+Wewv1nBrq7KWiEQAtwHTra7FaiLSEOgHzAAwxhQYY05ZW5WlvAF/EfEGAoB0i+upEa4W6OHA4RLLaXh4iAGISBTQDVhnbSWWmwT8FrBZXUgtEA1kALMcl6Cmi0ig1UVZwRhzBHgTOAQcBU4bY762tqqa4WqBrsoQkXrAR8CvjDFnrK7HKiJyO3DcGLPB6lpqCW+gOzDZGNMNyAY88jMnEQnC/k4+GmgOBIrIKGurqhmuFuhHgMgSyxGOdR5JRHywh/k8Y8wSq+uxWF9gsIikYr8Ud5OIzLW2JEulAWnGmPPv2hZjD3hPNBA4YIzJMMYUAkuA6yyuqUa4WqAnATEiEi0ivtg/2FhqcU2WEBHBfn10hzHmLavrsZox5gVjTIQxJgr738V/jTFu2QqrDGPMz8BhEWnnWDUASLGwJCsdAnqLSIDjdTMAN/2A2NvqAqrCGFMkIk8CX2H/pHqmMWa7xWVZpS/wAPCTiGx2rPudMWaZhTWp2uUpYJ6j8bMfGGNxPZYwxqwTkcXARuy9wzbhpkMA6Ff/lVLKTbjaJRellFIV0EBXSik3oYGulFJuQgNdKaXchAa6Ukq5CQ10pZRyExroSinlJv4/S6z/U3WZmF0AAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"code","metadata":{"id":"R-w1o3ErwOn9","executionInfo":{"status":"ok","timestamp":1605374880640,"user_tz":420,"elapsed":1060,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"4d5d6517-94f5-4452-93d0-2d0ee2fbc72f","colab":{"base_uri":"https://localhost:8080/"}},"source":["cnn_rand.load_weights('best_model.hdf5')\n","cnn_rand.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n","results = cnn_rand.evaluate(X_test, y_test)\n","print('Test accuracy: ', results[1])"],"execution_count":28,"outputs":[{"output_type":"stream","text":["67/67 [==============================] - 0s 5ms/step - loss: 1.3431 - accuracy: 0.4060\n","Test accuracy:  0.40600094199180603\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_3niNISAwOoA"},"source":["## CNN-static\n","- A model with pre-trained vectors from ```word2vec```. All words - including the unknown ones that are randomly initialized- are kept static and only the other parameters of the model are learned"]},{"cell_type":"code","metadata":{"id":"RvvAOoatwOoA","executionInfo":{"status":"ok","timestamp":1605374885383,"user_tz":420,"elapsed":488,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def cnn_static(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n","    inputs = Input(shape = (max_len, ))\n","    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n","                         input_length = max_len, weights = [embedding_matrix], trainable = False)\n","    embedded = embedd_intializer(inputs)\n","\n","    \n","    conv_results = []\n","    for window in windows:\n","        x = Conv1D(filters, window, activation = 'relu')(embedded)\n","        x = Dropout(p)(x)\n","        x = MaxPooling1D(max_len - window + 1)(x)\n","        conv_results.append(x)\n","    conv_result = concatenate(conv_results)\n","    x = GlobalMaxPooling1D()(conv_result)\n","    x = Dense(50)(x)\n","    outputs = Dense(5, activation = 'softmax')(x)\n","    m = Model(inputs = inputs, outputs = outputs)\n","    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n","    return m"],"execution_count":29,"outputs":[]},{"cell_type":"code","metadata":{"id":"dXUX-aaawOoD","executionInfo":{"status":"ok","timestamp":1605374888685,"user_tz":420,"elapsed":432,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["cnn_static = cnn_static(embed_dim, num_tokens, X_train.shape[1])\n","callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"],"execution_count":30,"outputs":[]},{"cell_type":"code","metadata":{"id":"V7hvTIuCwOoG","executionInfo":{"status":"ok","timestamp":1605374926764,"user_tz":420,"elapsed":31406,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"16a34c8a-ea9b-4aef-9c53-b16473a4539e","colab":{"base_uri":"https://localhost:8080/"}},"source":["history = cnn_static.fit(X_train, y_train, epochs = 10, callbacks = callbacks, validation_split = 0.1, batch_size = 256)"],"execution_count":32,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5653 - acc: 0.2889\n","Epoch 00001: val_acc improved from -inf to 0.33281, saving model to best_model.hdf5\n","23/23 [==============================] - 3s 135ms/step - loss: 1.5653 - acc: 0.2889 - val_loss: 1.5436 - val_acc: 0.3328\n","Epoch 2/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5407 - acc: 0.3002\n","Epoch 00002: val_acc did not improve from 0.33281\n","23/23 [==============================] - 3s 126ms/step - loss: 1.5407 - acc: 0.3002 - val_loss: 1.5438 - val_acc: 0.3328\n","Epoch 3/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5295 - acc: 0.3150\n","Epoch 00003: val_acc did not improve from 0.33281\n","23/23 [==============================] - 3s 128ms/step - loss: 1.5295 - acc: 0.3150 - val_loss: 1.5386 - val_acc: 0.3297\n","Epoch 4/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5224 - acc: 0.3231\n","Epoch 00004: val_acc improved from 0.33281 to 0.33906, saving model to best_model.hdf5\n","23/23 [==============================] - 3s 131ms/step - loss: 1.5224 - acc: 0.3231 - val_loss: 1.5209 - val_acc: 0.3391\n","Epoch 5/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5161 - acc: 0.3190\n","Epoch 00005: val_acc did not improve from 0.33906\n","23/23 [==============================] - 3s 126ms/step - loss: 1.5161 - acc: 0.3190 - val_loss: 1.5296 - val_acc: 0.3359\n","Epoch 6/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5052 - acc: 0.3341\n","Epoch 00006: val_acc improved from 0.33906 to 0.35781, saving model to best_model.hdf5\n","23/23 [==============================] - 3s 127ms/step - loss: 1.5052 - acc: 0.3341 - val_loss: 1.5147 - val_acc: 0.3578\n","Epoch 7/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4900 - acc: 0.3449\n","Epoch 00007: val_acc did not improve from 0.35781\n","23/23 [==============================] - 3s 127ms/step - loss: 1.4900 - acc: 0.3449 - val_loss: 1.5128 - val_acc: 0.3500\n","Epoch 8/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4773 - acc: 0.3598\n","Epoch 00008: val_acc did not improve from 0.35781\n","23/23 [==============================] - 3s 129ms/step - loss: 1.4773 - acc: 0.3598 - val_loss: 1.5244 - val_acc: 0.3297\n","Epoch 9/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4757 - acc: 0.3614\n","Epoch 00009: val_acc did not improve from 0.35781\n","23/23 [==============================] - 3s 129ms/step - loss: 1.4757 - acc: 0.3614 - val_loss: 1.5077 - val_acc: 0.3344\n","Epoch 10/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4672 - acc: 0.3688\n","Epoch 00010: val_acc did not improve from 0.35781\n","23/23 [==============================] - 3s 130ms/step - loss: 1.4672 - acc: 0.3688 - val_loss: 1.5020 - val_acc: 0.3187\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"0-Nc0IEqwOoI","executionInfo":{"status":"ok","timestamp":1605374926914,"user_tz":420,"elapsed":29551,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"f3411e75-db4d-4a50-8422-d107254fa6b3","colab":{"base_uri":"https://localhost:8080/","height":265}},"source":["plt.plot(history.history['acc'])\n","plt.plot(history.history['val_acc'])\n","plt.legend(['training', 'validation'], loc = 'upper left')\n","plt.show()"],"execution_count":33,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZfbA8e8hCQm99yQSihB6CQmKCgpqUARsIIqCiojCou66P9nVta2uru6y6oooAq6iiFhQVARBUcQChCIltFATaugtCSnn98ed4IABApnkzkzO53l4nNtmzozJyZ33nnteUVWMMcYErzJuB2CMMaZ4WaI3xpggZ4neGGOCnCV6Y4wJcpbojTEmyIW6HcCpatasqQ0bNnQ7DGOMCSiLFy/eo6q1Ctrmd4m+YcOGJCUluR2GMcYEFBHZcrptNnRjjDFBzhK9McYEuUIlehFJFJG1IpIiIqMK2D5MRFaIyDIRmS8iLTzrb/Osy/+XJyLtfP0mjDHGnJ6crQWCiIQA64ArgTRgETBAVZO99qmsqoc8j3sD96tq4inP0xr4VFUbn+n14uLi9NQx+uzsbNLS0sjMzCz0GzNnFhERQWRkJGFhYW6HYozxARFZrKpxBW0rzMXYeCBFVTd6nmwK0Ac4kejzk7xHBaCgvx4DgCmFDdpbWloalSpVomHDhojI+TyF8aKq7N27l7S0NGJiYtwOxxhTzAozdNMASPVaTvOsO4mIDBeRDcALwMgCnqc/8H5BLyAiQ0UkSUSS0tPTf7c9MzOTGjVqWJL3ERGhRo0a9g3JmFLCZxdjVXWMZ1jmEeAx720ikgAcU9WVpzl2nKrGqWpcrVoFloFakvcx+zyNKT0Kk+i3AVFey5GedaczBeh7yrpbOM3ZvDHGlHaqysyVO5iycGuxPH9hEv0ioKmIxIhIWZykPd17BxFp6rV4LbDea1sZoB/nOT7vLw4cOMBrr712zsddc801HDhw4Iz7PP7448yZM+d8QzPGBLCkzfu4cexPDHt3CR8kpVIcc4Sc9WKsquaIyAhgFhACTFTVVSLyNJCkqtOBESLSA8gG9gODvJ7iMiA1/2JuoMpP9Pfff/9J63NycggNPf3HOGPGjLM+99NPP13k+IwxgSVl9xFemLmGr5N3UbtSOP+8sTU3dogslmHVQrVAUNUZwIxT1j3u9fiBMxz7HdD5POPzG6NGjWLDhg20a9eOsLAwIiIiqFatGmvWrGHdunX07duX1NRUMjMzeeCBBxg6dCjwW0uHI0eO0LNnTy655BJ++uknGjRowGeffUa5cuUYPHgwvXr14qabbqJhw4YMGjSIzz//nOzsbD788EOaN29Oeno6t956K9u3b+eiiy5i9uzZLF68mJo1a7r8yRhjzsXuw5m8PGc9UxalUi4shIevupC7LomhfNni60jjd71uzuapz1eRvP3Q2Xc8By3qV+aJ61qecZ/nn3+elStXsmzZMr777juuvfZaVq5ceaI8ceLEiVSvXp2MjAw6derEjTfeSI0aNU56jvXr1/P+++/z5ptv0q9fPz7++GMGDhz4u9eqWbMmS5Ys4bXXXuNf//oX48eP56mnnuKKK67gL3/5CzNnzmTChAm++wCMMcXuaFYOb/6wkXHzNnI8J4+BCdH8oXtTalYML/bXDrhE7y/i4+NPqkF/5ZVXmDZtGgCpqamsX7/+d4k+JiaGdu2cG4M7duzI5s2bC3zuG2644cQ+n3zyCQDz588/8fyJiYlUq1bNp+/HGFM8snPz+GBRKi/NWc+eI1lc27oeD1/djJiaFUoshoBL9Gc78y4pFSr89j/pu+++Y86cOfz888+UL1+ebt26FVijHh7+21/ukJAQMjIyCnzu/P1CQkLIycnxceTGmJKgqnydvIt/zlzDxvSjxDeszrg7OtIhuuRP0qypWSFVqlSJw4cPF7jt4MGDVKtWjfLly7NmzRp++eUXn79+ly5dmDp1KgBff/01+/fv9/lrGGN8Y/GW/dz8+s/cO2kxArx5Rxwf3NvZlSQPAXhG75YaNWrQpUsXWrVqRbly5ahTp86JbYmJibz++uvExsbSrFkzOnf2/bXnJ554ggEDBjBp0iQuuugi6tatS6VKlXz+OsaY87cx/QgvzlrLVyt3UqtSOP+4vjX94iIJDXH3nPqsTc1KWkFNzVavXk1sbKxLEfmHrKwsQkJCCA0N5eeff+a+++5j2bJlRXpO+1yN8Y30w1m88s163l+4lfDQMtzbtTFDLi3eSppTFbWpmfEDW7dupV+/fuTl5VG2bFnefPNNt0MyptQ7djyH8T9s4o3vN5CZk8et8dGM7N6UWpWKv5LmXFiiDxBNmzZl6dKlbodhjAFycvP4cHEao2evI/1wFokt6/LnxGY0rlXR7dAKZIneGGMKSVWZs3o3/5y5hpTdR+h4QTVeH9iBjhdUdzu0M7JEb4wxhbB0636em7GGhZv30ahmBd64vSNXtagTEJ1gLdEbY8wZbN5zlBdnreXLFTuoWbEsz/RtRf9OUYS5XElzLizRG2NMAfYeyeK/36bw7i9bKBtahge6N+WeyxpRMTzw0mbg/EkKMBUrOhdltm/fzk033VTgPt26dePUUtJTvfTSSxw7duzEcmHaHhtjzl/G8VzGzE2h64vfMemXLfTrFMV3D3fjoSsvDMgkD3ZGX+zq16/PRx99dN7Hv/TSSwwcOJDy5csDhWt7bIw5d7l5ykeLUxk9ex27DmVxVYs6/F9ic5rU9s9KmnNhZ/SFNGrUKMaMGXNi+cknn+SZZ56he/fudOjQgdatW/PZZ5/97rjNmzfTqlUrADIyMrjllluIjY3l+uuvP6nXzX333UdcXBwtW7bkiSeeAJxGadu3b+fyyy/n8ssvB5y2x3v27AFg9OjRtGrVilatWvHSSy+deL3Y2FjuueceWrZsyVVXXXXanjrGGKeS5ts1u+j58jwe+XgF9auW48NhFzHujrigSPIQiGf0X42CnSt8+5x1W0PP58+4S//+/XnwwQcZPnw4AFOnTmXWrFmMHDmSypUrs2fPHjp37kzv3r1PexV+7NixlC9fntWrV7N8+XI6dOhwYtuzzz5L9erVyc3NpXv37ixfvpyRI0cyevRo5s6d+7u+84sXL+att95iwYIFqCoJCQl07dqVatWqFbodsjGlwbHjOWw/kMmOgxnsOJDJtgMZzuODmWw/kMH2A5lkZOfSsEZ5xt7WgcRWdQOikuZcFCrRi0gi8DLODFPjVfX5U7YPA4YDucARYKiqJnu2tQHeACoDeUAnVf19a0c/1759e3bv3s327dtJT0+nWrVq1K1bl4ceeoh58+ZRpkwZtm3bxq5du6hbt26BzzFv3jxGjhwJQJs2bWjTps2JbVOnTmXcuHHk5OSwY8cOkpOTT9p+qvnz53P99def6KJ5ww038MMPP9C7d+9Ct0M2JtBl5+ax05OwdxzMZLsnmW8/kMH2g05yP3As+3fH1aoUTv0qETStXYmuF9Ymtl4l+rZvEFCVNOfirIleREKAMcCVQBqwSESm5ydyj8mq+rpn/97AaCBRREKBd4HbVfVXEamBM93g+TvLmXdxuvnmm/noo4/YuXMn/fv357333iM9PZ3FixcTFhZGw4YNC2xPfDabNm3iX//6F4sWLaJatWoMHjz4vJ4nX2HbIRvjz/LylD1HspyEfSDDcybuJO/tnmSefiSLU9t1VSkXRr0qEdSvWo6OF1SlXpVy1K8aQb0q5WhQtRx1KkdQNjQ4E/rpFOaMPh5IyZ/zVUSmAH2AE4leVb2nfKoA5H/0VwHLVfVXz357fRG0W/r3788999zDnj17+P7775k6dSq1a9cmLCyMuXPnsmXLljMef9lllzF58mSuuOIKVq5cyfLlywE4dOgQFSpUoEqVKuzatYuvvvqKbt26Ab+1Rz516ObSSy9l8ODBjBo1ClVl2rRpTJo0qVjetzHFZf2uw2zdd+xEMvc+E995MJPs3JOzeLmwEOpVjaB+lXJ0vbAW9av+lsTz/1shQCtjilNhPpEGQKrXchqQcOpOIjIc+CNQFrjCs/pCQEVkFlALmKKqLxRw7FBgKEB0dPS5xF+iWrZsyeHDh2nQoAH16tXjtttu47rrrqN169bExcXRvHnzMx5/3333ceeddxIbG0tsbCwdO3YEoG3btrRv357mzZsTFRVFly5dThwzdOhQEhMTqV+/PnPnzj2xvkOHDgwePJj4+HgAhgwZQvv27W2YxgSM52as5o15G08sh5YR6lZxkniH6GqeM3Aneecn96rlw4Ju/LwknLVNsYjcBCSq6hDP8u1AgqqOOM3+twJXq+ogEXkYZ+y+E3AM+AZ4TFW/Od3rWZvikmOfq3HLG99v4Lmv1nBLpyj6d4qiftVy1KwYTkgZS+Lnq6htircBUV7LkZ51pzMFGOt5nAbMU9U9nkBmAB1wEr4xphSampTKc1+t4bq29fnH9a0pY8m92BXmisQioKmIxIhIWeAWYLr3DiLS1GvxWmC95/EsoLWIlPdcmO2K19i+MaZ0mZ28i798soJLm9bk3ze3tSRfQs56Rq+qOSIyAidphwATVXWViDwNJKnqdGCEiPTAqajZDwzyHLtfREbj/LFQYIaqfnk+gaqqjc35kL/NLFZqHNgK5WtA2Qpn3zfILNi4lxGTl9CqQRVeH9ix1FW+uKlQl6dVdQYw45R1j3s9fuAMx76LU2J53iIiIti7dy81atSwZO8DqsrevXuJiIhwO5TSJX0dvH4JlK8O3Z+ANv2hTOlIdsnbDzHknSQiq5XjrcGdrDKmhAXEpx0ZGUlaWhrp6eluhxI0IiIiiIyMdDuM0kMVvngIwiKgUj34dBgsfAMSn4do308m70+27j3GoLcWUjE8lHfuTqB6hbJuh1TqBESiDwsLIyYmxu0wjDl/yybDlvlw3SvQ/nZYMRXmPAkTr4ZWN0KPJ6Gq/5YWn6/0w1ncPnEB2bl5vH/PRTSoWs7tkEql0vG90Rg3Hd0LXz8G0Rc5Sb5MGWh7C/xhMXR9BNZ8Ca92gm+fgawjbkfrM4cysxk0cSG7D2Xx1uBONKldye2QSi1L9MYUt68fg6zD0Oulk8fky1aAy/8KI5KgeS+Y9yK8GgfL3oe8PPfi9YHM7FzueTuJdbsO8/rtHWkfXc3tkEo1S/TGFKdN8+DXydDlAah9mjunq0bBTRPgrq+hcn1n/H78FbD1l5KN1UdycvMY+f5SFm7ex7/7taXrhbXcDqnUs0RvTHHJznQuwFaLgcsePvv+0Qlw9xy4fhwc3uWM3390l1OSGSBUlUenreTr5F080asFfdo1cDskgyV6Y4rP/P/A3hToNRrCCnkRskwZaNsf/pAUkOP3L85aywdJqYy8ogmDu1gBhb+wRG9McUhfB/NHQ+t+0PiKs+9/Ku/x+9jrAmL8fvwPG3ntuw3cmhDNQ1de6HY4xoslemN87UTNfHm4+h9Fe66qUXDjeLh7tl+P309bmsYzX67mmtZ1+XufVnZjo5+xRG+Mr+XXzF/5NFT00YXIqHi/Hb+fu2Y3f/5wORc3rsF/+rezDpR+yBK9Mb50as28L/nh+P3iLfu4773FxNarzLg74ggPDXElDnNmluiN8aXT1cz7kp+M36/deZg731pEvSrleOvOTlS0/jV+yxK9Mb5SmJp5X3Jx/D5t/zHumLiAcmVDeOeueGpWDD/7QcY1luiN8YVzrZn3pRIev997JIs7Jiwk43gu79yVQFT18sXyOsZ3LNEb4wvnUzPvSyU0fn8kK4fBby1i+8EMJg7uRLO61r8mEFiiN6aoiloz70vFOH6flZPLvZOSSN5xiNdu60Bcw+o+CtoUt0IlehFJFJG1IpIiIqMK2D5MRFaIyDIRmS8iLTzrG4pIhmf9MhF53ddvwBhX+bJm3pd8PH6fm6c89MEyfkzZy4s3teGK5nV8HLApTmdN9CISAowBegItgAH5idzLZFVtrartgBeA0V7bNqhqO8+/Yb4K3Bi/UBw1877kg/F7VeVvn61kxoqdPHZtLDd0sAlrAk1hzujjgRRV3aiqx4EpQB/vHVT1kNdiBZz5YY0Jbvk181GdfV8z70unG7/f+H2hDv/PnPVMXrCV+7o1ZsiljYo5WFMcCpPoGwCpXstpnnUnEZHhIrIB54x+pNemGBFZKiLfi8ilBb2AiAwVkSQRSbLpAk3A+PoxyDoE1xVjzbwveY/fV4mCT++HzENnPOTtnzbzyjfr6R8Xxf9d3ayEAjW+5rOfTlUdo6qNgUeAxzyrdwDRqtoe+CMwWUQqF3DsOFWNU9W4WrX88OuvMac6qWY+1u1ozk3VKLj+dTi8Hb5+9LS7fbZsG09+voqrWtTh2eutf00gK0yi3wZEeS1HetadzhSgL4CqZqnqXs/jxcAGwNramcB2oma+IVz2Z7ejOT+Rcc4fqSXvwPrZv9v8/bp0/jT1V+IbVueVAe0JDQmAbyzmtArzf28R0FREYkSkLHALMN17BxFp6rV4LbDes76W52IuItIIaAps9EXgxrgmv2b+Wpdq5n2l21+gVixM/wNk7D+xeunW/dz37mKa1qnEm4PiiAiz/jWB7qyJXlVzgBHALGA1MFVVV4nI0yLS27PbCBFZJSLLcIZoBnnWXwYs96z/CBimqvt8/i6MKSknauZvhibd3Y6maELD4fqxcGQ3zPwrACm7D3PX/xZRq1I4b9/VicoRYS4HaXxBVP2rQCYuLk6TkpLcDsOY31OF//WCXSucC5oVa7sdkW98+yzMe4G9171Nr68rkZOnfDzsYqJrWGuDQCIii1U1rqBtNvBmTGGdVDMfJEke4LI/k1OrJXzxICGZ+3n7znhL8kHGEr0xhXFSzfwdbkfjU0dzy/DnnGFUzjvEp40+o0X93xXGmQBnid6Ywvj60cCqmS+k4zl5DHt3MZ/trMGWVsOpuWk6JE8/+4EmoATPT6wxxWXj9/Dr+4FZM38GeXnKnz78lR/W7+H5G9vQ5IbHoV5bp3T06B63wzM+ZInemDMJhpr5AqgqT32+is9/3c6ons3pFxcFIWHQ93Xnm8uXf3I7RONDluiNOZP5o2HfhsCvmT/Fq9+m8PbPW7jn0hjuvcyrf02dFk59ffKnsPJj9wI0PmWJ3pjTSV8HPwRJzbxHdm4eT05fxb9nr+OGDg34S8/Y37c2uHgkNOjonNUf3uVOoManLNEbUxBV+OJBKOtnfeaLYPfhTG57cwH/+2kzd3WJ4YUb21CmTAH9a0JCoe9YOH7MGbbys3ttzLmzRG9MQZa9B1t+DJqa+cVb9tHrlfks33aAl29px+PXtThz/5pazeCKx2Dtl7Diw5IL1BQLS/TGnOronqCpmVdVJv28mVvG/UK5siFMu78Lfdr9rst4wS4aDlEJMONhOLSjWOM0xcsSvTGn+voxyDoc8DXzmdm5/OnDX/nbZ6u4pElNpg+/hNh653AzVJkQ6PMa5ByHzx+wIZwAFrg/xcYUhyCpmU/dd4wbx/7EJ0u28WCPpkwY1Ikq5c+jQVnNJtDjCVg/y2kBYQJSqNsBGOM3gqRmft66dEZOWUpunjJhUBzdY4s4kXf8vbD6c5g5Chp1hSo2Z2ygsTN6Y/IFeM18Xp4yZm4Kg95aSN3KEXw+4pKiJ3lwhq/6jIG8XKd3vQ3hBBxL9MZAwNfMH87MZti7i3lx1lqua1OfT+6/mIY1K/juBarHwJVPwYZvYcnbvnteUyIKlehFJFFE1opIioiMKmD7MBFZISLLRGS+iLQ4ZXu0iBwRkYd9FbgxPhPgNfPrdx2mz5gf+WbNbv7WqwUv39KO8mWLYVQ27m6I6QqzHoX9W3z//KbYnDXRe6YCHAP0BFoAA05N5MBkVW2tqu2AF4DRp2wfDXzlg3iN8b0Arpn/asUO+o75kUMZ2bw3JIG7L4kpvkm8y5SBPq86j6ePgLy84nkd43OFOaOPB1JUdaOqHseZ/LuP9w6qeshrsQJwYhBPRPoCm4BVRQ/XGB8L0Jr5nNw8nvtqNfe9t4QL61biiz9cSudGNYr/hatGw9XPwqZ5kDSh+F/P+ERhEn0DINVrOc2z7iQiMlxENuCc0Y/0rKsIPAI8VfRQjSkGAVgzv/dIFoPeWsgb329kYOdopgztTN0qESUXQIdB0Lg7zH4c9m0sudc1581nP9mqOkZVG+Mk9sc8q58E/qOqR850rIgMFZEkEUlKT0/3VUjGnFkA1swvTzvAdf+dz6LN+3nhpjY807c14aEhJRuECPT+L5QJg0+H2xBOAChMot8GRHktR3rWnc4UoK/ncQLwgohsBh4E/ioiI049QFXHqWqcqsbVqlWrUIGbAJB12OmAOO5y+PJhWPERHNjqH+V5AVgzP3VRKje9/jMiwsfDLnZ6yLulSgNIfA62/gQL33AvDlMohbk0vwhoKiIxOAn+FuBW7x1EpKmqrvcsXgusB1DVS732eRI4oqqv+iBu4+82/wifDoODaRDZybmrctGbzrZK9SEqHqI7O/+t28aZ9KIk5dfMD/zE72vms3JyeXJ6Mu8v3MolTWryyoD2VK9Q1u2woN2tsHo6zHkKmlzp3EVr/NJZE72q5njOwmcBIcBEVV0lIk8DSao6HRghIj2AbGA/MKg4gzZ+LDsTvv07/DzGOVu+cyZEJ0BuDuxeBakLYesvzn+TP3WOCS3n9D+PTnCaaEV2gvLViy/GAKqZ33Ewg2HvLuHX1APc160xD1/VjJCCWgu7QQR6vQSvdYZP74O7Zjr9cYzfEfWHr9Fe4uLiNCkpye0wzPnYvhSmDYP0NU7N9ZVPQ3jF0+9/cBukLYStCyD1F9ixHDTX2Vaz2W+JP6oz1GjsJJaiUoX/XQu7VsKIJL8up/x5w15GTF5CZnYu/+7XlsRW9dwOqWDLp8In98CVf4cuI92OptQSkcWqGlfQNut1Y4ouN9s5Q573AlSoBQM/hiY9zn5clQZQ5Xpoeb2zfPwobFsCqQucf8nTYck7zrbyNTxJP95J/PXbQ9h5VJrk18xf97LfJnlVZcL8TTz31Roa1ijPG7d3pkntSm6HdXqtb4bkz+DbZ+DCq51e9sav2Bm9KZr0dTDtXti+BFr3g2tegHLVfPPceXmwZ91viT91AexNcbaVCYP67TzJ3/Ov0ln6uhzdA6/GOd8W7vzKL8spj2bl8MjHy/li+Q6ublmHf93clkoRJXz94nwc2Q1jEpzhurtnO7NUmRJ1pjN6S/Tm/OTlOdUWc56EsPLQa/RvZ+bF6egeZ3w/1TPOv20J5GY526o1PDnx1449ecx42jBntqRh8/2ynHLTnqPcOymJlN1H+PPVzRnWtVHx3eVaHFZ+Ah/dCd0fh0v/5HY0pY4N3RjfOrAVPr0fNv8AFybCda+c/WzaVyrUhObXOP8AcrKcsf3UX5wz/g1zYfkHzrbwyhAZ5wz1lKvm1Mxf+ie/TPJzknfx0AfLCA0R3r4rnkubBmCZcasbnCGcuc85Pxd1WrodkfGwM3pTeKrOGPdXowCFxOeh/UDfXCT1FVXYv/m3oZ6tC2B3MqDOGf/9v/hVOWVunvLynHW88m0KrRpU5vWBHYmsVt7tsM7f0b3wWgJUqgf3fFvyZbOlmJ3Rm6I7stuZTm7tDLigC/R9zUmc/kbEaalbPQba3uKsyzwI2xZD9UZ+leQPHsvmgQ+W8t3adG7qGMkzfVsRERbg5YkVakCv/8AHA50L9N0ecTsigyV6UxjJ0502vllH4KpnofP9fnkh87QiqkDjK9yO4iTJ2w8x7N3F7DiYwTN9W3FbQnRgjcefSex1TiXOvBegWU+o18btiEq9APptNSUu4wB8ci9MvR2qRMG98+DiEYGV5P3Qp0u3ccPYH8nKyeWDey9iYOcLgifJ5+v5glMS++l9zuTi/uzIbtiz/uz7BTA7ozcF2/AtfDYCDu+ErqPgsodtvLWI8vKUZ2esZsL8TcTHVGfMrR2oVSnc7bCKR/nqzr0K79/inNlf8djZjylJx/Y58+Cu/NgpKpAQ5/6PRl3djqxYWKI3Jzt+FGY/4fSlqXkhDJnttCcwRfbPmWuYMH8Tgy9uyKPXxhIWEuTfjJr1hLa3OmP1za6BBh3cjSfzkHONaeXHzolMXo5z3ebSP8HqL+CD2502DnVOnVcp8FnVjflN6iLn5qd9G6DzcOj+N7+6eBnIxv+wkWe+XM3tnS/g6T4tg2+o5nQyDsBrF0FEZWfoL7SEv8EcPwbrZsKqT2Dd1849F1WinHs+Wt0I9do6F/APpML4HlAmFIbMgcp+2m7iDEpH1U12hlP65w/KhDkXoOq0Dow7BHOOw/fPw/z/QOUGMOhziLnM7aiCxqdLt/HMl6u5pnVdnuxdipI8QLmqTu/6926Euf9wJhgvbjlZkPKNc+a+9ivIPgoV60DcndDyBqdp3qnXmapGwW1T4a1rYPLNzp3T4X7cduIcBUAWKqTjR53e5/4krLynK2Pn37oylqvqdlQn27XKueC6a4VTE3/1c87Zl/GJeevSefjDX+ncqDqj+7Xzn86TJalpD+hwB/z0CjTvBVGdfP8audmw6Xvn7tzVX0DWQShXHdr0c27kuqDL2Ttr1msLN78Nk/vB1EFw6wdBc10qeIZu8vLg2B7fB3Q+jh91er9s9dy0s3PFb10Za8V6dWVMcMYI3TjDy8t1fvHm/sMpP+z9X2dM1fjMr6kHGPDmL1xQowIf3NuZyoHQs6a4ZB6CsRdDaAQM+8E3Q4J5ubDlJ2dYJvkzOLbXuRu6eS9nWKZR1/NL1Ivfhs9HOn+crnvFv24IPIPSMXRTpox/dSOsHuP8sIFTf+6d+FdNg8X/c7ZVqHVyV8Z6bc+vK+O52LcRpt3ntA2Ivc7pKV6hZvG+ZimzMf0Id/5vEdUrlOXtOzuV7iQPzrfE3v+FSX2dLpdXP3t+z6MKaUnOsMyqaXBkp/PNuVlP5/etcfei//50HAQHU2Hei85k6AEyA9mZBE+i92fhFZ0x7/xx77w82LP2twk4Un+BNV8420LKOi148xN/VLzv/oCpQtJE+PpvzkWn68c5X20D5IwlUOw+lMkdExciwKS7E6hduQQn7vZnjS935in4eYxz1n3BRYU7ThV2LneS+8ppcHArhPTwMJcAABluSURBVIRD0yudYZkLE6FsBd/GevmjzgXab5+BKtHQtr9vn7+EFWroRkQSgZdxZpgar6rPn7J9GDAcyAWOAENVNVlE4oFx+bsBT6rqtDO9VqmtujmSfnI73u1LIddzo0n1Rid3ZazV/NxvWjq0Hab/AVLmQKPLoc8Ypx+88alDmdn0f+MXtuw9ypShnWkT6WfXZNyWdQTGXuScaAybf+YEvXuNJ7l/7FSClQl1fnZb3eg0tYuoUryx5hyHd29wTsgCoMa+SG2KRSQEWAdcCaThzCE7QFWTvfaprKqHPI97A/eraqKIlAeOe6YjrAf8CtRX1ZzTvV6pTfSnysmC7cu8mnP98ts1iPAqzgWt/DP+Bh1PP5OTqvOL8uUfnR/cq/4OnYbYWXwxyMzOZfBbC0navJ+Jgztx2YUB2IGyJGz6Ad7uBfH3OvMXeNu30bmguvITZ+pJKQMNL3GSe2zv4p1isiAZB2BionOi5Oc19kUdo48HUlR1o+fJpgB9gBOJPj/Je1QA1LP+mNf6iPz1phBCw52LttEJzrKq80vg3Yt97rOAOnf11W31W+KP7gxVIp27/778ozOWGRkP17/uTMlnfC43T3nog2X8snEfL/VvZ0n+TGIuhYRhsOB15xpR9RjnZ3Tlx843WXB+lnu+CC36lFwL7IKUqwq3fejU2L93c+DW2BfijP4mIFFVh3iWbwcSVHXEKfsNB/4IlAWuUNX1nvUJwETgAuD2goZuRGQoMBQgOjq645YtW4r6vkqHjAPOhan8Xuxpi52aYXDq4XOynM6Nl/8VujxgEzcXE1Xl8c9WMemXLTx2bSxDLm3kdkj+7/hReP0Sp8VGtud8sF4758y95fVOXbs/2fGrU2NfPcZva+yLOnRTqETvtf+twNWqOuiU9bHA28Blqpp5utezoZsiyM1xJr3OH+rJOgw9noC6rd2OLKi98s16Rs9ex72XNeIv1/jfpCZ+Ky3JabfRuJtzI5O/f9tcP8epsW/UzS9r7Is6dLMN8P7zGulZdzpTgLGnrlTV1SJyBGgFWCYvDiGhzjyq9dtBwr1uR1MqTF6wldGz13FDhwY8ktjc7XACS2Qc3Pml21EUXtMeTq/9z0c6Q6IBVGNfmNKNRUBTEYkRkbLALcB07x1EpKnX4rVA/rBNjIiEeh5fADQHNvsgbmNcN2vVTh77dAXdmtXinze2oUxpvOu1tOk4yKmrX/IO/PAvt6MptLOe0XsqZkYAs3DKKyeq6ioReRpIUtXpwAgR6QFkA/uB/GGbS4BRIpIN5OFU4/jJ7avGnL+Fm/bxh/eX0iayKq/d1iH4O1Ga3wRgjX3wtEAwpoSs2XmIfq//TM1K4Xw07GKqVyjrdkimpPlhjf2ZxujtNMSYc5C2/xiDJi6kXNkQ3rkr3pJ8aRVaFvq/CzWaOH3sdyWf/RgXWaI3ppD2HT3OHRMXcux4Lm/fFU9ktfJuh2TclF9jH1bOqbE/tMPtiE7LEr0xhXDseA53/W8RafszGH9HHM3rWitnw2997DMPOH3ssw67HVGBLNEbcxbZuXkMf28Jy9MO8N8B7UloVMPtkIw/ye9jvyvZ6WOfm+12RL9jid6YM1BVHvl4OXPXpvNM39Zc3bKu2yEZf5RfY7/hG6fG3s+KXKxNsTFn8PzMNXyyZBsP9biQWxOi3Q7H+DM/7mNvid6Y0xj/w0be+H4jAztHM7J7E7fDMYHAT2vsLdEbU4DPljkTevdsVZenercqXRN6m/Mn4sykdWgbfDYcKtX1ixp7G6M35hT5E3onxFTnP/1L6YTe5vz5YY29JXpjvCxPO8CwdxfTuFZF3hwUR0SYtXY258HPauwt0ZtilbR5Hx8s2sqOgxluh3JWm/Yc5c63PBN63xVvE3qbovGjGnsbozfFYtehTJ79cjXTf91+Yl2rBpXpEVuHK1vUoUW9yn417u1M6L0ABd65K546NqG38YX8GvvJ/Zwae5f62FtTM+NT2bl5vPXjJl6es57sPGVY18YktqzLvPXpzE7exZKt+1GF+lUi6NGiDj1i65DQqDrhoe4NkXhP6P3+PZ1pG2UTehsfW/y208e+wx3F1se+qBOPGFMoP6bs4Ynpq0jZfYTuzWvz+HUtuKBGBQBa1K/MsK6N2XMki2/X7GZO8i4+TErjnZ+3UDE8lK4X1qJHi9pc3qw2VcuXXKOwzOxchr6TxPpdh5kwuJMleVM8XK6xt0RvimzHwQye+WI1X67YQXT18kwYFEf32IIndK5ZMZx+cVH0i4siMzuXnzbsYXbybr5ZvYsvV+wgpIwQd0E1rvSc7TesWaHY4s7NU/449bcJvbvahN6mOLlYY29DN+a8Hc/JY/z8jfz3mxTyVLm/WxPu7drovCpV8vKUFdsOMmf1LmYn72LNTufCVZPaFU+M67eLquqzUkdV5Ynpq3jnZ5vQ25SgYuxjX6TJwT1PkAi8jDPD1HhVff6U7cOA4UAucAQYqqrJInIl8DxQFjgO/FlVvz3Ta1miDwzz1qXz5PRVbNxzlKta1OFvvVoQVd13bXtT9x1jzupdzFm9iwUb95GTp9SsWJYrmtemR2wdLmlak/Jlz/8L6X+/Wc+/bUJv44aMAzAx0bmp6q5ZUKeFT562SIleREKAdcCVQBrOHLIDVDXZa5/KqnrI87g3zpSBiSLSHtilqttFpBUwS1UbnOn1LNH7t20HMvj758nMXLWThjXK82TvlnRrVrtYX/NgRjbfr0tnTvIu5q7dzeHMHMJDy3BJk5r0aFGH7s1rU/scqmSmLNzKqE9WcEP7Bvzr5rY216speQdSYXwPKBMCQ+ZA5fpFfsqiXoyNB1JUdaPnyaYAfYATiT4/yXtUANSzfqnX+lVAOREJV9Wsc3sLxm1ZObm8OW8jr85NAeDPVzdjyKUxJVItU6VcGL3b1qd32/pk5+axaNM+ZnuGeL5ZsxuAtlFVuTK2Nj1a1KFZnUqnLd38etVO/jrNM6H3TTaht3FJfo39W9fAe/3grq8gvFKxvVxhzuhvAhJVdYhn+XYgQVVHnLLfcOCPOMM0V6jq+gKeZ5iq9ijgNYYCQwGio6M7btmy5fzfkfG5uWt289Tnq9i89xg9W9XlsV4taFC1nNthoaqs23WE2ck7mb16N7+mHgAgslo5esTW4aoWdegUU/3ExN2LNu9j4PgFNK9XmffvSSjS0I8xPrF+jlNj36hbkWvsizp0U6hE77X/rcDVqjrIa11LYDpwlapuONPr2dCN/0jdd4ynPk9mzupdNKpVgad6t+TSpv5bmbL7UCbfeEo356fsISsnj0oRoVzerDbxMdV5YeYam9Db+J/8Gvv2tzsN0c6zxr6oQzfbgCiv5UjPutOZAoz1evFIYBpwx9mSvPEPmdm5vP79BsZ+t4GQMsIjic25+5IYyob6d8eM2pUjGBAfzYD4aI4dz2H++j3MWb2Lb1bvZvqv26ldKdwm9Db+56Qa+wugq+9r7AuT6BcBTUUkBifB3wLc6r2DiDT1Gqq5FljvWV8V+BIYpao/+ixqU2zmJO/iqS9Wkbovg15t6vHotbHUq+L+MM25Kl82lKta1uWqlnXJzVOWpx2gTuUI6vvBkJMxv5NfY799CeTlOhdpfeisiV5Vc0RkBDALp7xyoqquEpGngSRVnQ6MEJEeQDawH8gfthkBNAEeF5HHPeuuUtXdPn0Xpsi27D3KU58n8+2a3TSpXZHJQxK4uElNt8PyiZAyQvvoam6HYczp5fexlzI+T/JgN0yVehnHcxn7XQqvz9tIWBnhwR4XMrhLwxMXMI0xgcF63ZjfUVVmrdrF379IZtuBDPq0q89fr4m1ro3GBCFL9KXQxvQjPPl5MvPWpdOsTiWmDO1M50Y13A7LGFNMLNGXIseO5/DqtymM/2ET4aFleLxXC26/6AIbpjEmyFmiLwVUla9W7uSZL5LZfjCTGzo0YFTP5tSuZMM0xpQGluiDXMruIzw5fRXzU/YQW68yLw9oT6eG1d0OyxhTgizRB6mM47m89M06Js7fRERYCE/1bsltCdGE2jCNMaWOJfoglJWTy5B3FvFjyl5u7hjJIz2bU7NiuNthGWNcYok+yOTk5vHA+8v4MWUv/765LTd2jHQ7JGOMy+x7fBBRVR6dtpKZq3byt14tLMkbYwBL9EHl+a/W8EFSKiOvaMLdl8S4HY4xxk9Yog8SY7/bwBvzNnLHRRfw0JUXuh2OMcaPWKIPAu8v3Mo/Z66hd9v6PHldy9POrmSMKZ0s0Qe4GSt28Khnarx/97P5T40xv2eJPoD9sD6dB6YspUN0Ncbe1tFaGRhjCmSZIUAt3bqfeyctpnGtikwY3IlyZYt/km5jTGCyRB+A1u06zOC3FlGrUjjv3B1PlXLnP6GwMSb4FSrRi0iiiKwVkRQRGVXA9mEiskJElonIfBFp4VlfQ0TmisgREXnV18GXRqn7jnH7hAWEh5bh3bsTrDGZMeaszproRSQEGAP0BFoAA/ITuZfJqtpaVdsBLwCjPeszgb8BD/su5NIr/XAWt09YQGZ2HpPuTiCqenm3QzLGBIDCnNHHAymqulFVjwNTgD7eO6jqIa/FCoB61h9V1fk4Cd8UwcGMbO6YuJBdh7KYOLgTzepWcjskY0yAKEyvmwZAqtdyGpBw6k4iMhz4I1AWuOJcghCRocBQgOjo6HM5tFTIOJ7LkLcXkbL7MBMGdaLjBTbRtTGm8Hx2MVZVx6hqY+AR4LFzPHacqsapalytWrV8FVJQyM7NY/jkJSRt2c9/+rfjsgvt8zHGnJvCJPptQJTXcqRn3elMAfoWJSjjyMtTHv7wV75ds5tn+7amV5v6bodkjAlAhUn0i4CmIhIjImWBW4Dp3juISFOvxWuB9b4LsXRSVZ76fBWfLdvOn69uxq0JNqRljDk/Zx2jV9UcERkBzAJCgImqukpEngaSVHU6MEJEegDZwH5gUP7xIrIZqAyUFZG+wFWqmuz7txJcXpqznrd/3sI9l8Zwf7fGbodjjAlghZp4RFVnADNOWfe41+MHznBsw/MNrrR668dNvPzNem7uGMlfr4m1JmXGmCKxO2P9zLSlaTz1eTJXtajDcze0tiRvjCkyS/R+5JvVu3j4w+Vc3LgGrwxobxN5G2N8wjKJn1iwcS/3v7eElvUrM+6OOCLCrEmZMcY3LNH7gZXbDjLk7SQiq5Xjf3fGUzHc5mw3xviOJXqXbdpzlMFvLaRSRCiT7k6geoWybodkjAkyluhdtONgBgPHL0AVJg1JoH7Vcm6HZIwJQjZG4JL9R49zx4SFHMzIZsrQzjSuVdHtkIwxQcoSvQuOZOUw+H+L2LLvGO/cFU+rBlXcDskYE8Rs6KaEZeXkcu+kJFZuO8iYWzvQuVENt0MyxgQ5S/QlKCc3jwfeX8aPKXt54cY2XNmijtshGWNKAUv0JURVeXTaSmau2snjvVpwY8dIt0MyxpQSluhLyPNfreGDpFRGXtGEuy6JcTscY0wpYom+BIz9bgNvzNvIHRddwENXXuh2OMaYUsYSfTF7f+FW/jlzDb3b1ufJ61pakzJjTImzRF+MZqzYwaPTVtCtWS3+3a8tZcpYkjfGlLxCJXoRSRSRtSKSIiKjCtg+TERWiMgyEZkvIi28tv3Fc9xaEbnal8H7sx/Wp/PAlKV0iK7G2Ns6EmadKI0xLjlr9hGREGAM0BNoAQzwTuQek1W1taq2A14ARnuObYEz9WBLIBF4zfN8QW3p1v3cO2kxjWtVZMLgTpQrG/Rv2RjjxwpzmhkPpKjqRlU9jjP5dx/vHVT1kNdiBUA9j/sAU1Q1S1U3ASme5wtay1IPcOf/FlGrUjjv3B1PlXJhbodkjCnlCpPoGwCpXstpnnUnEZHhIrIB54x+5DkeO1REkkQkKT09vbCx+51Pl26j3xs/UzE8lEl3JVC7UoTbIRljjO8uxqrqGFVtDDwCPHaOx45T1ThVjatVq5avQioxeXnKP2eu4cEPltE+qirTR1xCdI3ybodljDFA4ZqabQOivJYjPetOZwow9jyPDThHsnJ4cMpS5qzezYD4aJ7q3ZKyoXbh1RjjPwqTkRYBTUUkRkTK4lxcne69g4g09Vq8FljveTwduEVEwkUkBmgKLCx62P4hdd8xbnztJ+auTeep3i35x/WtLMkbY/zOWc/oVTVHREYAs4AQYKKqrhKRp4EkVZ0OjBCRHkA2sB8Y5Dl2lYhMBZKBHGC4quYW03spUb9s3Mt97y4mN095+854Lmla0+2QjDGmQKKqZ9+rBMXFxWlSUpLbYZzR5AVbefyzlUTXKM+EQZ2IqVnB7ZCMMaWciCxW1biCttnEI+cgJzePZ75czf9+2kzXC2vxyoD2Vj5pjPF7lugL6eCxbIZPXsL8lD0MuSSGv1wTS4i1NDDGBABL9IWQsvsI97yTRNr+Y7xwUxv6xUWd/SBjjPETlujP4ru1u/nD+0sJDy3D+/d0Jq5hdbdDMsaYc2KJ/jRUlQnzN/GPGatpVrcyb97RkchqdhOUMSbwWKIvQFZOLo9NW8mHi9NIbFmXf/drS4Vw+6iMMYHJstcp9hzJYtikxSRt2c/I7k15sHtT6yNvjAlolui9JG8/xD3vJLH3aBav3tqeXm3qux2SMcYUmSV6j5krd/LQB8uoUi6MD++9mNaRVdwOyRhjfKLUJ3pV5dVvU/j37HW0i6rKuNs7UruytRc2xgSPUp3oM47n8n8fL+fzX7dzffsGPHdDayLCbDYoY0xwKbWJfufBTO55J4mV2w/ySGJzhnVthIhddDXGBJ9SmeiXbt3P0EmLOZaVw5u3x9GjRR23QzLGmGJT6hL9p0u38X8fL6dO5XDevbsLzepWcjskY4wpVqUm0eflKS9+vZax320gIaY6Ywd2pHqFsm6HZYwxxa5UJHrv6f5uTYjmyetsuj9jTOlRqGwnIokislZEUkRkVAHb/ygiySKyXES+EZELvLb9U0RWev7192XwhbF17zFueO1H5q5N5+k+LXm2r033Z4wpXc56Ri8iIcAY4EogDVgkItNVNdlrt6VAnKoeE5H7gBeA/iJyLdABaAeEA9+JyFeqesjXb6Qg+dP95Sm8c1c8XZrYdH/GmNKnMKe28UCKqm5U1ePAFKCP9w6qOldVj3kWfwEiPY9bAPNUNUdVjwLLgUTfhH5mkxdsZeD4BVSvUJZPh3exJG+MKbUKk+gbAKley2medadzN/CV5/GvQKKIlBeRmsDlwO9m7RCRoSKSJCJJ6enphYv8NHJy83jis5X8ddoKujSpybThXWxOV2NMqebTi7EiMhCIA7oCqOrXItIJ+AlIB34Gck89TlXHAePAmRz8fF/fe7q/ey6NYVRPm+7PGGMKk+i3cfJZeKRn3UlEpAfwKNBVVbPy16vqs8Cznn0mA+uKEvDppO0/xu0TFrJtfwYv3tSGm226P2OMAQqX6BcBTUUkBifB3wLc6r2DiLQH3gASVXW31/oQoKqq7hWRNkAb4GtfBe+tZsVwYmpW4MWb2th0f8YY4+WsiV5Vc0RkBDALCAEmquoqEXkaSFLV6cCLQEXgQ0+/mK2q2hsIA37wrDsEDFTVnOJ4IxFhIUwc3Kk4ntoYYwJaocboVXUGMOOUdY97Pe5xmuMycSpvjDHGuMTuHDLGmCBnid4YY4KcJXpjjAlyluiNMSbIWaI3xpggZ4neGGOCnCV6Y4wJcqJ63q1lioWIpANbivAUNYE9Pgon0NlncTL7PE5mn8dvguGzuEBVaxW0we8SfVGJSJKqxrkdhz+wz+Jk9nmczD6P3wT7Z2FDN8YYE+Qs0RtjTJALxkQ/zu0A/Ih9Fiezz+Nk9nn8Jqg/i6AbozfGGHOyYDyjN8YY48USvTHGBLmgSfQikigia0UkRURGuR2Pm0QkSkTmikiyiKwSkQfcjsltIhIiIktF5Au3Y3GbiFQVkY9EZI2IrBaRi9yOyU0i8pDn92SliLwvIhFux+RrQZHoPVMWjgF64kx0MkBESvOEJznAn1S1BdAZGF7KPw+AB4DVbgfhJ14GZqpqc6AtpfhzEZEGwEggTlVb4cyid4u7UfleUCR6IB5IUdWNqnocmAL0cTkm16jqDlVd4nl8GOcXuYG7UblHRCKBa4HxbsfiNhGpAlwGTABQ1eOqesDdqFwXCpQTkVCgPLDd5Xh8LlgSfQMg1Ws5jVKc2LyJSEOgPbDA3Uhc9RLwf0Ce24H4gRggHXjLM5Q1XkQquB2UW1R1G/AvYCuwAzioql+7G5XvBUuiNwUQkYrAx8CDqnrI7XjcICK9gN2qutjtWPxEKNABGKuq7YGjQKm9piUi1XC+/ccA9YEKIjLQ3ah8L1gS/TYgyms50rOu1BKRMJwk/56qfuJ2PC7qAvQWkc04Q3pXiMi77obkqjQgTVXzv+F9hJP4S6sewCZVTVfVbOAT4GKXY/K5YEn0i4CmIhIjImVxLqZMdzkm14iI4IzBrlbV0W7H4yZV/YuqRqpqQ5yfi29VNejO2ApLVXcCqSLSzLOqO5DsYkhu2wp0FpHynt+b7gThxelQtwPwBVXNEZERwCycq+YTVXWVy2G5qQtwO7BCRJZ51v1VVWe4GJPxH38A3vOcFG0E7nQ5Hteo6gIR+QhYglOttpQgbIdgLRCMMSbIBcvQjTHGmNOwRG+MMUHOEr0xxgQ5S/TGGBPkLNEbY0yQs0RvjDFBzhK9McYEuf8HjzvRoXnB85YAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"code","metadata":{"id":"6sc3T4R0wOoK","executionInfo":{"status":"ok","timestamp":1605374928379,"user_tz":420,"elapsed":973,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"132bf3b8-1f4e-49cd-fba4-d7042dafb64c","colab":{"base_uri":"https://localhost:8080/"}},"source":["cnn_static.load_weights('best_model.hdf5')\n","cnn_static.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n","results = cnn_static.evaluate(X_test, y_test)\n","print('Test accuracy: ', results[1])"],"execution_count":34,"outputs":[{"output_type":"stream","text":["67/67 [==============================] - 0s 6ms/step - loss: 1.5273 - accuracy: 0.3310\n","Test accuracy:  0.33098921179771423\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"TqAC-C-1wOoN"},"source":["## CNN-non-static\n","- Same as above, but the pretrained vectors are fine-tuned for each task"]},{"cell_type":"code","metadata":{"id":"DgKw1I0owOoO","executionInfo":{"status":"ok","timestamp":1605374928520,"user_tz":420,"elapsed":1109,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def cnn_non_static(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n","    inputs = Input(shape = (max_len, ))\n","    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n","                     input_length = max_len, weights = [embedding_matrix], trainable = True)\n","    embedded = embedd_intializer(inputs)\n","    \n","    conv_results = []\n","    for window in windows:\n","        x = Conv1D(filters, window, activation = 'relu')(embedded)\n","        x = Dropout(p)(x)\n","        x = MaxPooling1D(max_len - window + 1)(x)\n","        conv_results.append(x)\n","    conv_result = concatenate(conv_results)\n","    x = GlobalMaxPooling1D()(conv_result)\n","    x = Dense(50)(x)\n","    outputs = Dense(5, activation = 'softmax')(x)\n","    m = Model(inputs = inputs, outputs = outputs)\n","    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n","    return m"],"execution_count":35,"outputs":[]},{"cell_type":"code","metadata":{"id":"cypFTYpfwOoQ","executionInfo":{"status":"ok","timestamp":1605374928522,"user_tz":420,"elapsed":1108,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["cnn_non_static = cnn_non_static(embed_dim, num_tokens, X_train.shape[1])\n","callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"],"execution_count":36,"outputs":[]},{"cell_type":"code","metadata":{"id":"AkjAb0kTwOoS","executionInfo":{"status":"ok","timestamp":1605374969135,"user_tz":420,"elapsed":40198,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"2dced6b4-1a26-4470-d7ca-45f1c80d76bc","colab":{"base_uri":"https://localhost:8080/"}},"source":["history = cnn_non_static.fit(X_train, y_train, epochs = 10, callbacks = callbacks, validation_split = 0.1, batch_size = 256)"],"execution_count":37,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5930 - acc: 0.2627\n","Epoch 00001: val_acc improved from -inf to 0.31406, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 173ms/step - loss: 1.5930 - acc: 0.2627 - val_loss: 1.5551 - val_acc: 0.3141\n","Epoch 2/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5325 - acc: 0.3075\n","Epoch 00002: val_acc did not improve from 0.31406\n","23/23 [==============================] - 4s 163ms/step - loss: 1.5325 - acc: 0.3075 - val_loss: 1.5457 - val_acc: 0.2641\n","Epoch 3/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4907 - acc: 0.3431\n","Epoch 00003: val_acc improved from 0.31406 to 0.35781, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 163ms/step - loss: 1.4907 - acc: 0.3431 - val_loss: 1.5157 - val_acc: 0.3578\n","Epoch 4/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4336 - acc: 0.3935\n","Epoch 00004: val_acc did not improve from 0.35781\n","23/23 [==============================] - 4s 160ms/step - loss: 1.4336 - acc: 0.3935 - val_loss: 1.4871 - val_acc: 0.3484\n","Epoch 5/10\n","23/23 [==============================] - ETA: 0s - loss: 1.3542 - acc: 0.4418\n","Epoch 00005: val_acc did not improve from 0.35781\n","23/23 [==============================] - 4s 162ms/step - loss: 1.3542 - acc: 0.4418 - val_loss: 1.4822 - val_acc: 0.3578\n","Epoch 6/10\n","23/23 [==============================] - ETA: 0s - loss: 1.2296 - acc: 0.5200\n","Epoch 00006: val_acc improved from 0.35781 to 0.40938, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 162ms/step - loss: 1.2296 - acc: 0.5200 - val_loss: 1.4160 - val_acc: 0.4094\n","Epoch 7/10\n","23/23 [==============================] - ETA: 0s - loss: 1.0452 - acc: 0.6206\n","Epoch 00007: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 161ms/step - loss: 1.0452 - acc: 0.6206 - val_loss: 1.3772 - val_acc: 0.3984\n","Epoch 8/10\n","23/23 [==============================] - ETA: 0s - loss: 0.8505 - acc: 0.7191\n","Epoch 00008: val_acc did not improve from 0.40938\n","23/23 [==============================] - 4s 162ms/step - loss: 0.8505 - acc: 0.7191 - val_loss: 1.3511 - val_acc: 0.4047\n","Epoch 9/10\n","23/23 [==============================] - ETA: 0s - loss: 0.6443 - acc: 0.7985\n","Epoch 00009: val_acc improved from 0.40938 to 0.41875, saving model to best_model.hdf5\n","23/23 [==============================] - 4s 163ms/step - loss: 0.6443 - acc: 0.7985 - val_loss: 1.3568 - val_acc: 0.4187\n","Epoch 10/10\n","23/23 [==============================] - ETA: 0s - loss: 0.4542 - acc: 0.8784\n","Epoch 00010: val_acc did not improve from 0.41875\n","23/23 [==============================] - 4s 161ms/step - loss: 0.4542 - acc: 0.8784 - val_loss: 1.3758 - val_acc: 0.4141\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"fQw03-cRwOoV","executionInfo":{"status":"ok","timestamp":1605374969619,"user_tz":420,"elapsed":40674,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"a808facb-64d0-43bf-e249-522a22a3555f","colab":{"base_uri":"https://localhost:8080/","height":266}},"source":["plt.plot(history.history['acc'])\n","plt.plot(history.history['val_acc'])\n","plt.legend(['training', 'validation'], loc = 'upper left')\n","plt.show()"],"execution_count":38,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD5CAYAAAA3Os7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deVyVZf7/8dcFgsgiIqCioGCiIriguGdpajlalmkuaWWT2mZOfav5VfOdalrmW43TMo3WuGVp7m2WNjZNmppLoLjihoiCqCAIguyc6/fHjQSIetQD9zmHz/Px4MG5l3PfH4/65ua67+u6lNYaIYQQjs/F7AKEEELYhgS6EEI4CQl0IYRwEhLoQgjhJCTQhRDCSUigCyGEk2hgzU5KqWHAB4ArME9r/Va17W2ABUAgkAVM0lqnXumYAQEBOjQ09HpqFkKIemvHjh1ntdaBNW27aqArpVyBWcBQIBWIVUqt1lonVNptJvCZ1vpTpdRtwP8BD1zpuKGhocTFxVn7ZxBCCAEopY5fbps1TS69gEStdZLWuhhYBtxdbZ9OwE/lr9fXsF0IIUQtsybQWwEplZZTy9dVthu4t/z1KMBHKeVf/UBKqWlKqTilVFxGRsb11CuEEOIybHVT9DngVqVUPHArcBIoq76T1nqO1jpGax0TGFhjE5AQQojrZM1N0ZNASKXl4PJ1FbTWaZRfoSulvIHRWuvsay2mpKSE1NRUCgsLr/WtogYeHh4EBwfj5uZmdilCiDpgTaDHAuFKqTCMIB8P3F95B6VUAJCltbYAL2I88XLNUlNT8fHxITQ0FKXU9RxClNNak5mZSWpqKmFhYWaXI4SoA1dtctFalwLTgXXAAWCF1nq/Uuo1pdTI8t0GAoeUUoeB5sCb11NMYWEh/v7+EuY2oJTC399fftsRoh6x6jl0rfVaYG21dS9Xer0KWGWLgiTMbUc+SyHqF+kpKoQQdaSotIy/rj1AWnZBrRxfAr2S7OxsZs+efc3vGz58ONnZV74H/PLLL/Pjjz9eb2lCCAd3IjOfMR9tZc7GJH46mF4r57CqyaW+uBjoTzzxRJX1paWlNGhw+Y9q7dq1l9120WuvvXbD9QkhHNO6/ad5buVuFDDngR7cHtmiVs4jV+iVvPDCCxw9epRu3brRs2dPBgwYwMiRI+nUqRMA99xzDz169CAyMpI5c+ZUvC80NJSzZ8+SnJxMREQEU6dOJTIykttvv52CAuNXq8mTJ7Nq1aqK/V955RW6d+9O586dOXjwIAAZGRkMHTqUyMhIpkyZQps2bTh79mwdfwpCCFspKbPwxncJPLpoB2EBXqyZMaDWwhzs+Ar9L9/uJyHtvE2P2allY165K/Ky29966y327dvHrl272LBhAyNGjGDfvn0Vj/0tWLCApk2bUlBQQM+ePRk9ejT+/lU7xB45coSlS5cyd+5cxo4dyxdffMGkSZMuOVdAQAA7d+5k9uzZzJw5k3nz5vGXv/yF2267jRdffJF///vfzJ8/36Z/fiFE3UnLLmD6kp3sPJHNQ33b8NKICBo2cK3Vc9ptoNuDXr16VXmG+x//+AdfffUVACkpKRw5cuSSQA8LC6Nbt24A9OjRg+Tk5BqPfe+991bs8+WXXwKwefPmiuMPGzYMPz8/m/55hBB1Y8OhdJ5ZvoviUgsfTojmrq4t6+S8dhvoV7qSriteXl4Vrzds2MCPP/7I1q1b8fT0ZODAgTU+492wYcOK166urhVNLpfbz9XVldLSUhtXLoQwQ2mZhfd/PMKsDYl0aO7D7IndaRvoXWfnlzb0Snx8fMjNza1xW05ODn5+fnh6enLw4EG2bdtm8/P379+fFStWAPDDDz9w7tw5m59DCFE70nMLmTR/O/9cn8jYHiF8/WT/Og1zsOMrdDP4+/vTv39/oqKiaNSoEc2bN6/YNmzYMD7++GMiIiLo0KEDffr0sfn5X3nlFSZMmMCiRYvo27cvLVq0wMfHx+bnEULY1pajZ5mxdBd5RSXMvK8rY3oEm1KH0lqbcuKYmBhdfYKLAwcOEBERYUo99qCoqAhXV1caNGjA1q1befzxx9m1a9cNHbO+f6ZC1CaLRTN7QyLv/ucwoQFefDSxBx1a1O5FmFJqh9Y6pqZtcoVuR06cOMHYsWOxWCy4u7szd+5cs0sSQlxG1oVinlm+i58PZzCya0v+797OeDU0N1Il0O1IeHg48fHxZpchhLiKHcezmL4knsy8Yt4cFcX9vVrbxdhJEuhCCGElrTXzNx/jre8P0rJJI758oh9RrXzNLquCBLoQQlghp6CE51fu5oeEM9wR2Zx3xnTFt5F9TR4jgS6EEFexNzWHJ5bs4FR2IX++sxO/72+fk/BIoAshxGVorVm8/QSvf5uAv7c7yx/tS4829tuDWzoW3QBvb6PTQFpaGmPGjKlxn4EDB1L98czq3n//ffLz8yuWrRmOVwhRu/KKSpmxbBd//nof/dr5s2bGALsOc5BAt4mWLVtWjKR4PaoH+tq1a2nSpIktShNCXIeDp88z8sPNrNmTxvN3dGDBQz1p6uVudllXJYFeyQsvvMCsWbMqll999VXeeOMNBg8eXDHU7TfffHPJ+5KTk4mKigKgoKCA8ePHExERwahRo6qM5fL4448TExNDZGQkr7zyCmAM+JWWlsagQYMYNGgQ8NtwvADvvvsuUVFRREVF8f7771ec73LD9AohbszKuBTumfULuUWlfD6lD08OaoeLi/21l9fEftvQv38BTu+17TFbdIbfvXXZzePGjePpp5/mySefBGDFihWsW7eOGTNm0LhxY86ePUufPn0YOXLkZW+IfPTRR3h6enLgwAH27NlD9+7dK7a9+eabNG3alLKyMgYPHsyePXuYMWMG7777LuvXrycgIKDKsXbs2MEnn3zC9u3b0VrTu3dvbr31Vvz8/KweplcIYZ2C4jJe/mYfK3ek0retPx9M6EYzHw+zy7omcoVeSXR0NOnp6aSlpbF79278/Pxo0aIFL730El26dGHIkCGcPHmSM2fOXPYYGzdurAjWLl260KVLl4ptK1asoHv37kRHR7N//34SEhKuWM/mzZsZNWoUXl5eeHt7c++997Jp0ybA+mF6hRBXdzQjj1Gzf2HVzlRm3NaOxVN6O1yYgz1foV/hSro23XfffaxatYrTp08zbtw4Pv/8czIyMtixYwdubm6EhobWOGzu1Rw7doyZM2cSGxuLn58fkydPvq7jXGTtML1CiCv7dncaL3yxh4Zurix8uBe3tg80u6TrJlfo1YwbN45ly5axatUq7rvvPnJycmjWrBlubm6sX7+e48ePX/H9t9xyC0uWLAFg37597NmzB4Dz58/j5eWFr68vZ86c4fvvv694z+WG7R0wYABff/01+fn5XLhwga+++ooBAwbY8E8rRP1VVFrGn7/ex1NL4+kY1Jg1M2526DAHK6/QlVLDgA8AV2Ce1vqtattbA58CTcr3eUFrffWZk+1QZGQkubm5tGrViqCgICZOnMhdd91F586diYmJoWPHjld8/+OPP87DDz9MREQEERER9OjRA4CuXbsSHR1Nx44dCQkJoX///hXvmTZtGsOGDaNly5asX7++Yn337t2ZPHkyvXr1AmDKlClER0dL84oQN+hEZj5PLtnJ3pM5TLulLc/f0QE3V8e/vr3q8LlKKVfgMDAUSAVigQla64RK+8wB4rXWHymlOgFrtdahVzquDJ9bN+QzFaKqH/af5tmVu1HAzPu61uqkzbXhRofP7QUkaq2Tyg+2DLgbqHxHTwONy1/7AmnXX64QQtheSZmFt78/yLzNx+jcypfZE7sT0tTT7LJsyppAbwWkVFpOBXpX2+dV4Ael1FOAFzCkpgMppaYB0wBat259rbUKIcR1Scsu4Kml8ew4fo4H+7bhTyMiaNjA1eyybM5WjUYTgIVa62BgOLBIKXXJsbXWc7TWMVrrmMDAmm8+mDWDkjOSz1IIWH8wnRH/2MTBU+f5cEI0r90d5ZRhDtZdoZ8EQiotB5evq+wRYBiA1nqrUsoDCADSr6UYDw8PMjMz8ff3t8uRzByJ1prMzEw8PBzvWVohbKG0zMLf/3OYjzYcpWMLH2ZP7F7nkzbXNWsCPRYIV0qFYQT5eOD+avucAAYDC5VSEYAHkHGtxQQHB5OamkpGxjW/VdTAw8OD4GBzJqsVwkyncwqZsTSeX5OzmNCrNa/c1QkPN+e8Kq/sqoGutS5VSk0H1mE8krhAa71fKfUaEKe1Xg08C8xVSj2DcYN0sr6O3/fd3NwICwu71rcJIUSFjYczeHr5LgpLynh/XDfuiW5ldkl1xqrn0MufKV9bbd3LlV4nAP2rv08IIepKmUXz/o+H+ef6RNo382HWxO60a+bcTSzV2W/XfyGEsFL6+UJmLItnW1IWY2OC+cvIKBq5O38TS3US6EIIh/ZL4ln+sCyevKJSZt7XlTE96u99Iwl0IYRDKrNoPvzpCB/89wg3BXqzZGof2jf3MbssU0mgCyEcTkZuEU8vj+eXxEzujW7FG6Oi8HSXOJNPQAjhULYezWTGsnjOF5Tw9ujOjI0JkX4r5STQhRAOwWLRzN6QyLv/OUxogBeLHulFxxaNr/7GekQCXQhh9zLzinhmxW42Hs7g7m4teXNUZ7wbSnxVJ5+IEMKu/Xosi6eW7uRcfgl/HdWZCb2kieVyJNCFEHbJYtH8a2MSM384RIhfIxY80ZPIlr5ml2XXJNCFEHbn3IVi/mfFLtYfymBElyDeurczPh5uZpdl9yTQhRB2ZcfxLKYviSczr5jX745kUp820sRiJQl0IYRd0Fozb9Mx3v73QVo2acQXj/ejc7A0sVwLCXQhhOmy84t5buVufjyQzrDIFrxzXxcaSxPLNZNAF0KYKv7EOaYviSc9t5BX7urE5H6h0sRynSTQhRCm0Fqz4Jdk3vr+AM0be7DysX50C2lidlkOTQJdCFHncgpK+OOq3azbf4ahnZozc0xXfD2lieVGSaALIerUntRsnlyyk1PZhfzviAgeuTlMmlhsRAJdCFEntNZ8tvU4b645QKBPQ1Y81pfurf3MLsupSKALIWrd+cISXvxiL2v2nmJwx2b8fWxXmni6m12W05FAF0LUqv1pOTzx+U5SzxXw4u86MnVAW1xcpImlNkigCyFqzdajmUz5NBYfDzeWT+tDTGhTs0tyahLoQoha8WPCGZ5YspM2TT1ZPKU3zRt7mF2S05NAF0LY3NfxJ3l25W6iWjZm4cO98POS9vK64GLNTkqpYUqpQ0qpRKXUCzVsf08ptav867BSKtv2pQohHMGirck8s2IXvUKb8vnUPhLmdeiqV+hKKVdgFjAUSAVilVKrtdYJF/fRWj9Taf+ngOhaqFUIYce01szecJS/rTvEkIjm/PP+aDzcXM0uq16x5gq9F5CotU7SWhcDy4C7r7D/BGCpLYoTQjgGrTX/9/1B/rbuEKOiW/HRpO4S5iawpg29FZBSaTkV6F3TjkqpNkAY8NONlyaEcARlFs2fvtrLstgUHuzbhlfvipTHEk1i65ui44FVWuuymjYqpaYB0wBat25t41MLIepacamFZ1bsYs2eU0wf1I5nb28v3fhNZE2Ty0kgpNJycPm6moznCs0tWus5WusYrXVMYGCg9VUKIexOQXEZUz+LY82eU/xpeATP3dFBwtxk1lyhxwLhSqkwjCAfD9xffSelVEfAD9hq0wqFEHYnp6CERxbGsvPEOd4e3ZlxPeU3bntw1UDXWpcqpaYD6wBXYIHWer9S6jUgTmu9unzX8cAyrbWuvXKFEGY7m1fEg/N/5Uh6Lh9O6M6ILkFmlyTKWdWGrrVeC6yttu7lasuv2q4sIYQ9OpldwAPztpOWU8DcB2MY2KGZ2SWJSqSnqBDCKkkZeUyat53colIWPdKbnjIui92RQBdCXNW+kzk8tOBXAJZN60NkS1+TKxI1kUAXQlxRXHIWDy+MxadhAxZP6U3bQG+zSxKXIYEuhLisDYfSeWzxDlr6NmLRlN60atLI7JLEFUigCyFqtGbPKZ5eHk94Mx8+e6QXAd4NzS5JXIUEuhDiEst+PcFLX+2le2s/5k/uiW8jN7NLElaQQBdCVDF3YxJvrj3Are0D+XhSDxq5yyBbjkICXQgBGCMm/v2Hw/xzfSIjOgfx3rhuuDewasoEYSck0IUQWCyaV7/dz2dbjzO+ZwhvjuqMq4yY6HAk0IWo50rKLPxx1R6+ij/JtFva8uLvOsogWw5KAl2IeqywpIzpS+L58cAZnr+jA08MvEnC3IFJoAtRT+UVlTL10zi2JmXy+t2RPNA31OySxA2SQBeiHjp3oZjJn/zKvrTzvDeuK6Oig80uSdiABLoQ9cyZ84VMmred41n5fDypB0M7NTe7JGEjEuhC1CMnMvOZOH8bWXnFLHy4J/1uCjC7JGFDEuhC1BOHTufywPztFJdZWDK1D11DmphdkrAxCXQh6oH4E+eY/EksDRu4sOLRvrRv7mN2SaIWSKAL4eS2JJ5lymdxBHg3ZPEjvWnt72l2SaKWSKAL4cR+2H+a6UvjCfP3YtEjvWjW2MPskkQtkkAXwkl9FZ/Kcyv30LmVLwsf7kkTT3ezSxK1TAJdCCf0+fbj/O/X++gT5s/ch2Lwbij/1esD+VsWwsnM25TEG2sOcFvHZsye2B0PNxn+tr6QQBfCSWit+fCnRN79z2GGd27B++OiZfjbekYCXQgnoLXm7X8f4uOfj3Jv91a8M7oLDVwlzOsbq/7GlVLDlFKHlFKJSqkXLrPPWKVUglJqv1JqiW3LFEJcjsWieXX1fj7++SgTe7dm5piuEub11FWv0JVSrsAsYCiQCsQqpVZrrRMq7RMOvAj011qfU0o1q62ChRC/KbNoXvhiDyt3pDJ1QBgvDY+Q4W/rMWuaXHoBiVrrJACl1DLgbiCh0j5TgVla63MAWut0WxcqhKiqpMzCM8t38d2eU/xhcDhPDwmXMK/nrPm9rBWQUmk5tXxdZe2B9kqpX5RS25RSw2o6kFJqmlIqTikVl5GRcX0VCyEoLCnj8cU7+G7PKV4a3pFnhraXMBfWtaFboQEQDgwEJgBzlVKXjPyjtZ6jtY7RWscEBgba6NRC1C/5xaVM+TSOHw+k8/o9UUy75SazSxJ2wpoml5NASKXl4PJ1laUC27XWJcAxpdRhjICPtUmVQggAzheW8PtPYtl54hwz7+vKmB4yMYX4jTVX6LFAuFIqTCnlDowHVlfb52uMq3OUUgEYTTBJNqxTiHrv3IViJs3bzq6UbD6c0F3CXFziqlfoWutSpdR0YB3gCizQWu9XSr0GxGmtV5dvu10plQCUAc9rrTNrs3Ah6pP03EIemPcrxzIv8K8HejA4QmYZEpdSWmtTThwTE6Pj4uJMObcQjiQtu4BJ87ZzKqeQeQ/F0L+dzDJUnymldmitY2raJj1FhbBjxzMvcP/c7ZwvKGHRI72ICW1qdknCjkmgC2GnEtNzmThvO0WlxpRxnYN9zS5J2DkJdCHs0P60HB6c/ytKKZZP60uHFjJlnLg6GfBBCDsTf+IcE+Zso2EDF1Y+JmEurCdX6ELYkW1JmTyyMJYAn4Z8PqU3wX4y/6ewngS6EHZiw6F0Hl20g9ZNPfl8Sm+Z/1NcMwl0IezAv/ed5qmlO2nf3IfPft8Lf++GZpckHJAEuhAm+zr+JM+u3E3XYF8+ebgXvo3czC5JOCgJdCFMtPTXE7z01V76hPkz76EYvGQyZ3ED5F+PECaZv/kYr3+XwKAOgXw0qYdM5ixumAS6ECb4509HmPnDYX4X1YIPxstkzsI2JNCFqENaa/627hCzNxxlVHQr/jZGJnMWtiOBLkQdsVg0r32XwMItydzfuzVv3B2Fi4vMMiRsRwJdiDpQZtG89OVelselMOXmMP40QiZzFrYngS5ELSsps/Dsit2s3p3GjMHhPCOTOYtaIoEuRC0qKi1j+pJ4/pNwhhd+15HHbpX5P0XtkUAXopYUFJcxbVEcm46c5bW7I3mwb6jZJQknJ4EuRC3ILSzhkYVxxB3P4m9junBfTMjV3yTEDZJAF8LGsi4U8/Anv7I/7TwfjI/mrq4tzS5J1BMS6ELY0PakTGYsi+dcfgkfT+rBkE4ymbOoOxLoQthAmUUze30i7/14mDb+Xsx/qCdRrWTKOFG3JNCFuEHpuYU8s3wXvyRmcne3lrw5qjPeMsiWMIH8qxPiBmw+cpanl8eTV1TKO6O7cF9MsDxjLkxj1SASSqlhSqlDSqlEpdQLNWyfrJTKUErtKv+aYvtShbAfpWUWZq47xAMLtuPn6c7q6TcztmeIhLkw1VWv0JVSrsAsYCiQCsQqpVZrrROq7bpcaz29FmoUwq6cyilgxtJ4YpPPMS4mhFdHRtLIXYa+FeazpsmlF5CotU4CUEotA+4Gqge6EE7vp4NneHbFbopLLbw/rhv3RLcyuyQhKljT5NIKSKm0nFq+rrrRSqk9SqlVSqkae1EopaYppeKUUnEZGRnXUa4Q5igutfDmmgR+vzCOIN9GfPvUzRLmwu7YaiDmb4FQrXUX4D/ApzXtpLWeo7WO0VrHBAYG2ujUQtSulKx87vvXVuZuOsaDfdvw5RP9aBvobXZZQlzCmiaXk0DlK+7g8nUVtNaZlRbnAe/ceGlCmO/7vaf44xd7APhoYnd+1znI5IqEuDxrAj0WCFdKhWEE+Xjg/so7KKWCtNanyhdHAgdsWqUQdaywpIy/rj3AZ1uP0zWkCf+cEE1IU0+zyxLiiq4a6FrrUqXUdGAd4Aos0FrvV0q9BsRprVcDM5RSI4FSIAuYXIs1C1GrkjLymL4knoRT55k6IIzn7+goc34Kh6C01qacOCYmRsfFxZlybiEu55tdJ3npy724NXDh7/d1ZXCEjMUi7ItSaofWOqambdJTVAiMsctfXb2f5XEp9Az14x8TognybWR2WUJcEwl0Ue8dPpPLk5/vJDEjj+mD2vH0kHAauEoTi3A8Euii3tJaszIulZdX78O7YQM++30vBoTL47TCcUmgi3opr6iU//1qL1/vSqN/O3/eG9eNZj4eZpclxA2RQBf1zv60HKYvied45gWeu709jw9sh6uLDKolHJ8Euqg3tNYs3nac19ccoKmnO0un9qF3W3+zyxLCZiTQRb2QU1DCC1/s4ft9pxnUIZC/j+1GUy93s8sSwqYk0IXT25WSzfQlOzmdU8ifhkfwyM1huEgTi3BCEujCaWmtmb/5GG99f5DmjT1Y+Vhfolv7mV2WELVGAl04pXMXinlu5W7+ezCdYZEteHt0F3w93cwuS4haJYEunE5schYzlsaTmVfMa3dH8kCfNjI1nKgXJNCF0ygqLWPuxiTe+/EIIX6N+PKJfkS18jW7LCHqjAS6cHg5+SUs3n6chVuSycgt4q6uLfnrqCh8PKSJRdQvEujCYZ3MLmDB5mMs+/UEF4rLGBAewHtju9G/nb80sYh6SQJdOJwDp84zZ2MS3+5OQwN3dQli6i1tiWwpzSuifpNAFw5Ba82Wo5n8a2MSGw9n4OnuyoN9Q/n9zaEE+8lMQkKABLqwc6VlFtbuO82cjUfZd/I8Ad4Nef6ODkzq3UYeQxSiGgl0YZfyi0tZEZvCvM3HSD1XQNsAL966tzP3RLfCw83V7PKEsEsS6MKunM0r4tMtySzadpzs/BJi2vjx8p2dGBLRXLrrC3EVEujCLhw7e4G5m5L4YkcqxWUWhkY059Fb29KjTVOzSxOOxGKBvDNw7hicS672dRwspeDu9duXm2e1197g7ln1tbsXuHlVe12+7OYFDexnkDcJdGGq+BPn+NfPSaxLOI2bqwuju7diyoC23BTobXZpwl4V50P28d+COqtSeGcfh9LCSjsr8A0Gv1AIHwKu7sb7i/OgJB+KL0B+FpRcMF4X5xuvtcX6elzcysPeu/wHQeXXNfzgcPeCtgOheaTtPpNyEuiizlksmp8OpjNnYxK/JmfR2KMBTw5sx0P9Qgn0aWh2ecJsFVfZydW+yoM770zV/d29wS8MAsIhfCg0DTMC3C8MfEOu/Qpaa+OHQpXgr/ZDoPhC+eu88m0Xyn8o5P+2Lf8sZJ+oul9ZkXGOO9+XQBeOrai0jG/i05izKYnE9DxaNWnEy3d2YlzPELwaOsk/xewU2PweHFoLjZqCTwtoHAQ+Lctfl3/3aQleAeBST2/wFucbYVdj00jypVfZjVsZQR0+9LewvvjdsynYsiOZUuDWyPjysvEEKGWlRvC71k4zjVX/i5RSw4APAFdgntb6rcvsNxpYBfTUWsfZrErh0HIKSliy/QSf/HKM9NwiOgU15oPx3RjeOQg3Vxezy7ONrGOw+V3YtdRY7jgcykog9xSc2Q8X0i/9NV65lod70KVhX7EcBA19bBtYtUFr40q08DwUnYfCnPLX5d/zzlRtGsk7XfX97t5GQPu3g3ZDqoZ2kxBo4CS/ubk2ANfa6wB31UBXSrkCs4ChQCoQq5RarbVOqLafD/AHYHttFCocT9rFrvmxKeQVlTIgPIC/j+3Kze0CnKdr/tlE2PR32LMcXBpAj8lw89NGu21lZaVGqJ8/ZYT8xa/zpyA3DTIT4dgmIwCrc/Mqv8oPqhb+5cuNg8C7xfXfnLvYxFAljHPKX5//7fsl66rtp8uucJLyq2y/0EqBHfpb84inv/3/0HIA1lyh9wIStdZJAEqpZcDdQEK1/V4H3gaet2mFwuEcPH2eOT8nsdqZu+anH4CNM2H/l+DaEHo/Bv2eMsK1Jq4NjBBu3PLKxy2+ALmnq4Z97mk4X/49ZZvxvaz40vd6BvwW8BfDvqEPFOVWCt6aQvk8WEqu8gdW0LAxeDT+7btPEAR2AA/fatt8L13n6Q9uHlZ9tOL6WRPorYCUSsupQO/KOyilugMhWus1SqnLBrpSahowDaB169bXXq2wW1prtiZl8q+fk/jZmbvmn94LG/8GCauNJxf6PQV9nwLvQNsc390L/G8yvi5Ha+PJjOphX3k5LR4uZFQ6rnfVkPUKNM5REcCVwviSdY3B3QdcnKR5zInd8J0opZQL8C4w+Wr7aq3nAHMAYmJi9I2eW5jPYtH8kHCa2RuOsic1p6Jr/l+9QHQAAA+lSURBVMTerWniaT/P596wkzuNID+01gi5Ac9Cnydsf9PMGkoZ5/XyhxadL79fabFxA65h4/p787WesSbQTwIhlZaDy9dd5ANEARvK20VbAKuVUiPlxqjzKimz8O3uNGZvOEpieh6h/p78372dGeVsXfNTfoWf34HE/4BHExj4EvR+FBo1Mbuyq2vgbledXkTtsybQY4FwpVQYRpCPB+6/uFFrnQMEXFxWSm0AnpMwd06FJWWs2pHKxz8fJfVcAR1b+PDhhGiGdw7C1Zm65if/AhvfgaQNRvvv4Jeh51Sj+UEIO3XVQNdalyqlpgPrMB5bXKC13q+Ueg2I01qvru0ihfkuFJXy+fbjzN10jIzcIqJbN+EvIyO5rWMz53liRWs49rNxRX78F/BqBre/ATG/N9q2hbBzVrWha63XAmurrXv5MvsOvPGyhL3Izi9m4ZZkFm5JJju/hP7t/PlgfDf6tnWiWYG0hsQfjSBP/dV4DnzY29DjIaNziRAOwkm65wlbS88tZP6mYyzedpwLxWUMiWjOk4NuIrq1n9ml2Y7WcOh7o2klLd7oJj7iXYie5DwdWUS9IoEuqkjJymfOxiSWx6VQWmbhzi4teWLQTXRs4URtxxYLHFhtPEd+Zq/RsWXkh9BlvNxEFA5NAl0AkJiex0cbjvLNrpMoBWN6BPPoLTcRGuBEbceWMtj/lRHkGQeMbuaj/gVRY4yOP0I4OPlXXM/tO5nD7A2JfL/vNA0buPBg31Cm3hJGkK8TtR2XlcLelbBpptHFPjACRs+HyFHyfLZwKhLo9VRschaz1iey4VAGPg2N4Wsf7h+Kv7cTtR2XFsPupcagWeeSoXlnGPsZdLxLej0KpySBXo9ordl45Cyzfkrk1+Qs/L3cef6ODjzQtw2NPZxowuXSIohfBJvfh5wUaNkdhr0F7YfJAFDCqUmgO7rT+4xmBK9A8G5mfPfwrRJcF7vnz1p/lL0ncwjy9eCVuzoxvmdrGrk7eJNDabExZsmFdLhwFtITYNtHxuBWIb2NiQTaDZYgF/WCBLoj270MvnnSmCexMld38GqGxSuA9LLG7DrnRlKBF3d4BvBSn/bERHbErXEBlJyDBk3tr/mhON8I6LyLQZ3x2+u88uULGcbrwuxL39/mZuNmZ9gtEuSiXpFAd0RaG+3C/30NQgfA7a9Dwbny0MugNPcMx48nk3E6Be/SE/RwzeV2t/O4FJfALoyvi5SLMezqxav7i98rXjczRhK8uM71OppmtDaGab1wtmooVw/niyFecqHm43j4ltfTDJpFQNitl9br0wKayEieon6SQHc0ljL4/o8QO8943O6e2RWdYCq658cdIyO3D91CmjB9UDs6dWyGi8II/SoBWilIL5w1XmcdNUK1tKDm8zfyM0LVK7A86MsD3zMASgoqXVlnVH19cS7FKpQxTsrFUA7ueeUfJtLZR4grkkB3JCUF8MUUOPgd9JsBQ/4CLi41d88f142+N1Xrnu/Z1PgK7HDl82htTLRQudkjL/3SK+xTe4x1lWfZcWlQNZQDIyqFcrNKPwSaGXNuyvPfQtiM/G9yFPlZsHS8MZzrsLfQvR8jPiWb1bvSWBmXUtE9/4lBN9H9RrvnKwUNvY2vpm2vvn9JoTHDuZuncQUv7dZCmEIC3RFkn4DFo9Hnkjl+2yyWZHVnzdvrOZldgLurC8OiWvD4wJuICDKpe76bx6VzaAoh6pwEup3Tp3ZTumgMZUUFPN/gz3y7tglurscYEB7Is7e3Z0in5s71DLkQ4rpJoNshrTWHzuSyZ+M3jEh4nhzdiEdK/0yzm6J5Z2gQd3Rqga+nhLgQoioJdDty5Ewu3+05xXd70ojKXMff3P7FGbcQdgyYy5KYbvh5yUiAQojLk0A3WVJGHt/tOcWaPac4dCYXpTRvBPzERPf5FIf0J+T+JYQ4wvyVQgjTSaCb4HjmhfIr8VMcOHUepaBnm6a8dldHxpydhWf8fIi8F/dRH8uz10IIq0mg15GUrHzW7jVCfO9J47nt7q2b8PKdnRjeOYgWnsCXU42JF/pOh6Gv21+XfCGEXZNAr0Vp2QUVIb4rxRhzpGuwL38aHsHwLkG0alI+5nh+FiyaCCe2wB1/hb5Pmli1EMJROV6gZ6fAqd3QYbhdXsGeOV9YEeI7jp8DIKpVY/7fsI7c2SWIkKaeVd+QnQKLR8O5YzBmAUSNNqFqIYQzcLxA37HQmHkmsCPc/IwRgNczYJQNZeQW8e99p/h2zylik7PQGjq28OG529szoktLwi43jdvpffD5GGN0wUlfQtiAui1cCOFUlNbalBPHxMTouLi4a39jWSkkfA2b/m6Mfe3bGvrPMGZqd6u7adOy84tZs9d4OmVbUiYWDeHNvLmzS0tGdAmiXTPvKx8g6WdYPgncvWHSKmgeWTeFCyEcmlJqh9Y6psZt1gS6UmoY8AHgCszTWr9VbftjwJNAGZAHTNNaJ1zpmNcd6BdZLHBknRHsqbHGgE99n4CYR8Cj9rrAJ6Sd59MtyXy96yRFpRbaBnhxZ5cg7uzakvbNfaw7yN5V8NVjxiTFk1ZJt3khhNVuKNCVUq7AYWAokArEAhMqB7ZSqrHW+nz565HAE1rrYVc67g0H+kVaQ/JmY3zwoz9BQ1/oNRX6PA5eATd+fKCkzMIP+8/w6ZZkfk3OwsPNhVHRwUzs3ZrIlo2rjmh4tVq3/hN++F9o0x/Gf24MZiWEEFa6UqBb04beC0jUWieVH2wZcDdQEegXw7ycF1B37ThKGW3PYQPg5E7Y/J5x1b51FvR4CPo9dd1XwGfzilj26wkWbzvB6fOFhDRtxJ+GRzA2JuTau95bLLDuJdj+EXS6x5hRx83juuoSQoiaWBPorYCUSsupQO/qOymlngT+B3AHbqvpQEqpacA0gNata2FWmVbdYdwiyDgMv7xvTAIROw+6jIebn4aAcKsOsyc1m4Vbkvlu9ymKyywMCA/gjXuiGNSxGa4u1zE0bEkhfPWo0fbf+3Hj0UQ7fEJHCOHYrGlyGQMM01pPKV9+AOittZ5+mf3vB+7QWj90peParMnlSrJPwJZ/ws5PjZngO42Em/8HWna7ZNfiUgvf7zvFwi3JxJ/IxsvdldE9gnmwb+jVb3BeScE5WDYRjv8Ct79hdBqS8cKFENfpRptcTgIhlZaDy9ddzjLgI+vLq0VNWsPwd+CW52HbbONqPeEbuGkwDHgW2vQjPbeIxdtPsGT7Cc7mFREW4MUrd3ViTI9gfG50WNqcVFg8BjITYfR86DzGNn8uIYSogTWBHguEK6XCMIJ8PHB/5R2UUuFa6yPliyOAI9gT70AY8orR7BI7H711FmrhcJI8ovhr7u/4r6UbA9s346F+odwSHojL9TSrVHdmvxHmxXkw6Qtoe+uNH1MIIa7gqoGutS5VSk0H1mE8trhAa71fKfUaEKe1Xg1MV0oNAUqAc8AVm1vMUujqzbeN7mOZR2cic77lMdYwz+1vFPt3wj3mWWjXHWwR5sc2Gc0s7p7w8PfQIurGjymEEFfheB2LrkNadgGLtx1nWWwKWReKCW/mzUP9QhnVpRleh78ynow5exj8woyr+K4Trn+Uw31fGM+Y+4UZV+ZNQq7+HiGEsNINdyyqDbUd6Fprth/L4tMtyfyQcAatNUMimjO5Xyh9b/Kv+uy4xQKH1hiPO6bFg0+QMUBWj4eNiZKttXWW8Whi674wfgl4NrX9H0wIUa/Vq0AvKC7j610n+XRLMgdP59LE041xPUOY1LvNpQNjVac1JK2HTe9C8iaj00+vR6H3o1cOZ4sF/vNno9NQxEi4d648Yy6EqBX1ItBTsvL5bGsyy2NTOF9YSkRQYyb3a8PIrq1o5O56HQeMNXqfHloLbl4Q87DxyGHjoKr7lRYZTSz7v4Re02DYW+ByHecTQggrOG2ga63ZnHiWT7ck89+D6bgoxbDIFjzUL5SeoX7Wd8m/kjP7YfP7sG8VuDSAbvdD/z9A07ZQkG0MsJW8CYb8xVgvz5gLIWqR0wV6XlEpX+5M5dMtyRzNuIC/lzsTerVmYp/WBPnW0oiLWcdgyz8gfjFYSiFyFKQfNG6m3jMbuoytnfMKIUQlThXoy2NP8MZ3B8gtKqVrsC8P9QtleOcgPNzqqJkj97Rx8zNuAaCMoQZuGlQ35xZC1Hs32lPUrrRq4sngCKMTUHRrE0Yq9GkBt79u9DQtLQKf5nVfgxBC1MDhAv3m8ABuDrfNsLg3pFETsysQQogqZMg/IYRwEhLoQgjhJCTQhRDCSUigCyGEk5BAF0IIJyGBLoQQTkICXQghnIQEuhBCOAnTuv4rpTKA49f59gDgrA3LcXTyeVQln8dv5LOoyhk+jzZa68CaNpgW6DdCKRV3ubEM6iP5PKqSz+M38llU5eyfhzS5CCGEk5BAF0IIJ+GogT7H7ALsjHweVcnn8Rv5LKpy6s/DIdvQhRBCXMpRr9CFEEJUI4EuhBBOwuECXSk1TCl1SCmVqJR6wex6zKKUClFKrVdKJSil9iul/mB2TfZAKeWqlIpXSn1ndi1mU0o1UUqtUkodVEodUEr1Nbsmsyilnin/f7JPKbVUKeVhdk21waECXSnlCswCfgd0AiYopTqZW5VpSoFntdadgD7Ak/X4s6jsD8ABs4uwEx8A/9ZadwS6Uk8/F6VUK2AGEKO1jgJcgfHmVlU7HCrQgV5AotY6SWtdDCwD7ja5JlNorU9prXeWv87F+M/aytyqzKWUCgZGAPPMrsVsSilf4BZgPoDWulhrnW1uVaZqADRSSjUAPIE0k+upFY4W6K2AlErLqdTzEANQSoUC0cB2cysx3fvAHwGL2YXYgTAgA/ikvAlqnlLKy+yizKC1PgnMBE4Ap4AcrfUP5lZVOxwt0EU1Silv4Avgaa31ebPrMYtS6k4gXWu9w+xa7EQDoDvwkdY6GrgA1Mt7TkopP4zf5MOAloCXUmqSuVXVDkcL9JNASKXl4PJ19ZJSyg0jzD/XWn9pdj0m6w+MVEolYzTF3aaUWmxuSaZKBVK11hd/a1uFEfD10RDgmNY6Q2tdAnwJ9DO5plrhaIEeC4QrpcKUUu4YNzZWm1yTKZRSCqN99IDW+l2z6zGb1vpFrXWw1joU49/FT1prp7wKs4bW+jSQopTqUL5qMJBgYklmOgH0UUp5lv+/GYyT3iBuYHYB10JrXaqUmg6sw7hTvUBrvd/ksszSH3gA2KuU2lW+7iWt9VoTaxL25Sng8/KLnyTgYZPrMYXWertSahWwE+PpsHicdAgA6fovhBBOwtGaXIQQQlyGBLoQQjgJCXQhhHASEuhCCOEkJNCFEMJJSKALIYSTkEAXQggn8f8Bf37FVfmKJT8AAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"code","metadata":{"id":"mPrwksxTwOob","executionInfo":{"status":"ok","timestamp":1605374971054,"user_tz":420,"elapsed":1069,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"73964e56-3ced-460e-be95-eafd8a1001af","colab":{"base_uri":"https://localhost:8080/"}},"source":["cnn_non_static.load_weights('best_model.hdf5')\n","cnn_non_static.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n","results = cnn_non_static.evaluate(X_test, y_test)\n","print('Test accuracy: ', results[1])"],"execution_count":39,"outputs":[{"output_type":"stream","text":["67/67 [==============================] - 0s 6ms/step - loss: 1.3634 - accuracy: 0.4037\n","Test accuracy:  0.40365681052207947\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ut4WzaHZwOof"},"source":["## CNN-multichannel\n","- A model with two sets of word vectors"]},{"cell_type":"code","metadata":{"id":"xM2eWVUPwOof","executionInfo":{"status":"ok","timestamp":1605374971055,"user_tz":420,"elapsed":1067,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["def cnn_multichannel(embed_dim, num_tokens, max_len, windows = [3, 4, 5], filters = 100, p = 0.5, l2 = 3):\n","    inputs = Input(shape = (max_len, ))\n","    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n","                     input_length = max_len, weights = [embedding_matrix], trainable = True)\n","    embedded_1 = embedd_intializer(inputs)\n","    \n","    embedd_intializer = Embedding(input_dim = len(word_idx) + 1, output_dim = embed_dim, \\\n","                     input_length = max_len, weights = [embedding_matrix], trainable = False)\n","    embedded_2 = embedd_intializer(inputs)\n","    \n","    embedded = concatenate([embedded_1, embedded_2])\n","    \n","    conv_results = []\n","    for window in windows:\n","        x = Conv1D(filters, window, activation = 'relu')(embedded)\n","        x = Dropout(p)(x)\n","        x = MaxPooling1D(max_len - window + 1)(x)\n","        conv_results.append(x)\n","    conv_result = concatenate(conv_results)\n","    x = GlobalMaxPooling1D()(conv_result)\n","    x = Dense(50)(x)\n","    outputs = Dense(5, activation = 'softmax')(x)\n","    m = Model(inputs = inputs, outputs = outputs)\n","    m.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['acc'])\n","    return m"],"execution_count":40,"outputs":[]},{"cell_type":"code","metadata":{"id":"IS6SlA4NwOoi","executionInfo":{"status":"ok","timestamp":1605374971213,"user_tz":420,"elapsed":1208,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}}},"source":["cnn_multichannel = cnn_multichannel(embed_dim, num_tokens, X_train.shape[1])\n","callbacks = [ModelCheckpoint(filepath = 'best_model.hdf5', monitor='val_acc', verbose=1, save_best_only = True, mode='max')]"],"execution_count":41,"outputs":[]},{"cell_type":"code","metadata":{"id":"WvvntknYwOok","executionInfo":{"status":"ok","timestamp":1605375032686,"user_tz":420,"elapsed":62677,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"975dde10-b699-4fe5-858c-61175afa4a81","colab":{"base_uri":"https://localhost:8080/"}},"source":["history = cnn_multichannel.fit(X_train, y_train, epochs = 10, callbacks = callbacks, validation_split = 0.1, batch_size = 256)"],"execution_count":42,"outputs":[{"output_type":"stream","text":["Epoch 1/10\n","23/23 [==============================] - ETA: 0s - loss: 1.6136 - acc: 0.2670\n","Epoch 00001: val_acc improved from -inf to 0.32031, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 257ms/step - loss: 1.6136 - acc: 0.2670 - val_loss: 1.5382 - val_acc: 0.3203\n","Epoch 2/10\n","23/23 [==============================] - ETA: 0s - loss: 1.5339 - acc: 0.3181\n","Epoch 00002: val_acc improved from 0.32031 to 0.34375, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 251ms/step - loss: 1.5339 - acc: 0.3181 - val_loss: 1.5302 - val_acc: 0.3438\n","Epoch 3/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4832 - acc: 0.3555\n","Epoch 00003: val_acc improved from 0.34375 to 0.35313, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 253ms/step - loss: 1.4832 - acc: 0.3555 - val_loss: 1.5024 - val_acc: 0.3531\n","Epoch 4/10\n","23/23 [==============================] - ETA: 0s - loss: 1.4312 - acc: 0.3846\n","Epoch 00004: val_acc improved from 0.35313 to 0.35938, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 253ms/step - loss: 1.4312 - acc: 0.3846 - val_loss: 1.4985 - val_acc: 0.3594\n","Epoch 5/10\n","23/23 [==============================] - ETA: 0s - loss: 1.3628 - acc: 0.4359\n","Epoch 00005: val_acc did not improve from 0.35938\n","23/23 [==============================] - 6s 252ms/step - loss: 1.3628 - acc: 0.4359 - val_loss: 1.4736 - val_acc: 0.3328\n","Epoch 6/10\n","23/23 [==============================] - ETA: 0s - loss: 1.2501 - acc: 0.4925\n","Epoch 00006: val_acc improved from 0.35938 to 0.41875, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 253ms/step - loss: 1.2501 - acc: 0.4925 - val_loss: 1.4092 - val_acc: 0.4187\n","Epoch 7/10\n","23/23 [==============================] - ETA: 0s - loss: 1.1045 - acc: 0.5561\n","Epoch 00007: val_acc improved from 0.41875 to 0.42344, saving model to best_model.hdf5\n","23/23 [==============================] - 6s 252ms/step - loss: 1.1045 - acc: 0.5561 - val_loss: 1.3800 - val_acc: 0.4234\n","Epoch 8/10\n","23/23 [==============================] - ETA: 0s - loss: 0.9179 - acc: 0.6472\n","Epoch 00008: val_acc did not improve from 0.42344\n","23/23 [==============================] - 6s 250ms/step - loss: 0.9179 - acc: 0.6472 - val_loss: 1.3659 - val_acc: 0.4047\n","Epoch 9/10\n","23/23 [==============================] - ETA: 0s - loss: 0.7539 - acc: 0.7316\n","Epoch 00009: val_acc did not improve from 0.42344\n","23/23 [==============================] - 6s 251ms/step - loss: 0.7539 - acc: 0.7316 - val_loss: 1.3580 - val_acc: 0.4000\n","Epoch 10/10\n","23/23 [==============================] - ETA: 0s - loss: 0.6076 - acc: 0.7903\n","Epoch 00010: val_acc did not improve from 0.42344\n","23/23 [==============================] - 6s 252ms/step - loss: 0.6076 - acc: 0.7903 - val_loss: 1.3741 - val_acc: 0.4141\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"3QFh-Ds3wOom","executionInfo":{"status":"ok","timestamp":1605375032849,"user_tz":420,"elapsed":62824,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"e51cbaf1-4923-48f3-c1ab-5c053eda99ed","colab":{"base_uri":"https://localhost:8080/","height":265}},"source":["plt.plot(history.history['acc'])\n","plt.plot(history.history['val_acc'])\n","plt.legend(['training', 'validation'], loc = 'upper left')\n","plt.show()"],"execution_count":43,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUVf7H8fdJhUACIQmEFJIAAUJCD01EQUQiKgqiFBVxBXZdXdeyBf3t2lZ3WddVd1fUBexSRGzoUmwoFkACUlLoBFIIKaT3yZzfHzeQIQQIMMmdmXxfzzNP5paZ+82QfLg599xzlNYaIYQQzs/N7AKEEELYhwS6EEK4CAl0IYRwERLoQgjhIiTQhRDCRXiYdeDAwEAdGRlp1uGFEMIpbdu2LU9rHdTYNtMCPTIyksTERLMOL4QQTkkpdeRs26TJRQghXESTAl0plaCU2quUOqCUmt/I9m5KqQ1KqZ+VUruUUhPtX6oQQohzOW+gK6XcgYXAtUBfYIZSqm+D3f4ErNRaDwKmAy/bu1AhhBDn1pQ29GHAAa31IQCl1ArgRiDFZh8N+NU97wBkXUwxNTU1ZGRkUFlZeTEvFw20adOGsLAwPD09zS5FCNECmhLooUC6zXIGMLzBPk8AnyulfgO0A66+mGIyMjLw9fUlMjISpdTFvIWoo7UmPz+fjIwMoqKizC5HCNEC7HVRdAbwptY6DJgIvKOUOuO9lVLzlFKJSqnE3NzcM96ksrKSgIAACXM7UEoREBAgf+0I0Yo0JdAzgXCb5bC6dbbuBlYCaK03AW2AwIZvpLVepLWO11rHBwU12o1SwtyO5LMUonVpSqBvBaKVUlFKKS+Mi56rG+xzFBgHoJSKwQj0M0/BhRCiFauoruVva1PJLKxolvc/b6BrrS3AfcB6IBWjN0uyUuoppdSkut0eBuYqpXYCy4HZ2gkHWi8sLOTlly+8g87EiRMpLCw85z6PPfYYX3755cWWJoRwcpsO5pPwr43899tDfL0np1mOoczK3fj4eN3wTtHU1FRiYmJMqQcgLS2N66+/nqSkpNPWWywWPDxMu6n2kpj9mQrR2pVWWViwNpV3Nx+lWycf/n5zf0b2CLjo91NKbdNaxze2Te4UtTF//nwOHjzIwIEDGTp0KKNHj2bSpEn07Wt0u7/pppsYMmQIsbGxLFq06NTrIiMjycvLIy0tjZiYGObOnUtsbCzXXHMNFRXGn1azZ89m1apVp/Z//PHHGTx4MP369WPPnj0A5ObmMn78eGJjY5kzZw4RERHk5eW18KcghLCXb/flMuGFjSzdcpS7L49i3QOjLynMz8dhTzuf/DSZlKxiu75n3xA/Hr8h9qzbFyxYQFJSEjt27OCbb77huuuuIykp6VS3v9dff51OnTpRUVHB0KFDufnmmwkIOP0fZ//+/SxfvpzFixdz66238sEHH3D77befcazAwEC2b9/Oyy+/zHPPPceSJUt48sknueqqq3jkkUdYt24dr732ml2/fyFEyygqr+Hp/6Xw/rYMegS1Y9WvLmNIhH+zH9dhA90RDBs27LQ+3P/+97/56KOPAEhPT2f//v1nBHpUVBQDBw4EYMiQIaSlpTX63lOmTDm1z4cffgjA999/f+r9ExIS8Pdv/h8AIYR9fZFynP/7aDf5ZdXcO7YHv7kqmjae7i1ybIcN9HOdSbeUdu3anXr+zTff8OWXX7Jp0yZ8fHwYM2ZMo328vb29Tz13d3c/1eRytv3c3d2xWCx2rlwI0dLyS6t48tMUVu/Mok+wL6/PHkpcaIcWrUHa0G34+vpSUlLS6LaioiL8/f3x8fFhz549bN682e7HHzVqFCtXrgTg888/p6CgwO7HEELYl9aaT3dmMf6FjaxNOsZD43ux+r7LWzzMwYHP0M0QEBDAqFGjiIuLo23btnTp0uXUtoSEBF599VViYmLo3bs3I0aMsPvxH3/8cWbMmME777zDyJEjCQ4OxtfX1+7HEULYR05xJX/+JIn1yccZENaBZ6eOoHeweb+z0m3RgVRVVeHu7o6HhwebNm3innvuYceOHZf0nq39MxWiOWit+WB7Jk99mkyVxcpD43tx9+VReLg3f6PHubotyhm6Azl69Ci33norVqsVLy8vFi9ebHZJQogGMgsrePTD3Xy7L5ehkf78/eb+dA9qb3ZZgAS6Q4mOjubnn382uwwhRCOsVs3yrUf525o9WLXmyUmx3DEiAjc3xxkzSQJdCCHO42h+OX/8YBebDuUzqmcAC6b0J7yTj9llnUECXQghzqLWqnnrxzT+sX4vHm6KBVP6MW1ouMOOZCqBLoQQjTiQU8ofP9jFtiMFjO0dxF+n9KNrh7Zml3VOEuhCCGHDUmtl8XeHeeHLffh4ufPCtAHcNDDUYc/KbcmNRZegfXvjynZWVhZTp05tdJ8xY8bQsHtmQy+++CLl5eWnlpsyHK8Qwv5SjxUz+eUf+fu6PYzr05kvHrySyYPCnCLMQQLdLkJCQk6NpHgxGgb6mjVr6Nixoz1KE0I0QbXFygtf7OOG/3zPsaIKXr5tMK/cPoQgX+/zv9iBSKDbmD9/PgsXLjy1/MQTT/D0008zbty4U0PdfvLJJ2e8Li0tjbi4OAAqKiqYPn06MTExTJ48+bSxXO655x7i4+OJjY3l8ccfB4wBv7Kyshg7dixjx44F6ofjBXj++eeJi4sjLi6OF1988dTxzjZMrxDiwuzKKGTSS9/zr6/2c8OAEL548Eom9utqdlkXxXHb0NfOh+zd9n3P4H5w7YKzbp42bRoPPPAA9957LwArV65k/fr13H///fj5+ZGXl8eIESOYNGnSWf8Ee+WVV/Dx8SE1NZVdu3YxePDgU9ueeeYZOnXqRG1tLePGjWPXrl3cf//9PP/882zYsIHAwNOnYd22bRtvvPEGW7ZsQWvN8OHDufLKK/H392/yML1CiMZV1tTy4pf7WbTxIEG+3rx2ZzzjYrqc/4UOzHED3QSDBg0iJyeHrKwscnNz8ff3Jzg4mAcffJCNGzfi5uZGZmYmx48fJzg4uNH32LhxI/fffz8A/fv3p3///qe2rVy5kkWLFmGxWDh27BgpKSmnbW/o+++/Z/LkyadGfZwyZQrfffcdkyZNavIwvUKIMyWmneAPH+ziUG4Z04eG88jEGDq09TS7rEvmuIF+jjPp5nTLLbewatUqsrOzmTZtGkuXLiU3N5dt27bh6elJZGRko8Pmns/hw4d57rnn2Lp1K/7+/syePfui3uekpg7TK4SoV15t4R/r9/Lmj2mEdGjLO3cPY3R0kNll2Y20oTcwbdo0VqxYwapVq7jlllsoKiqic+fOeHp6smHDBo4cOXLO119xxRUsW7YMgKSkJHbt2gVAcXEx7dq1o0OHDhw/fpy1a9eees3Zhu0dPXo0H3/8MeXl5ZSVlfHRRx8xevRoO363QrQemw7mM+HFjbzxQxqzRkTw+YNXuFSYgyOfoZskNjaWkpISQkND6dq1K7fddhs33HAD/fr1Iz4+nj59+pzz9ffccw933XUXMTExxMTEMGTIEAAGDBjAoEGD6NOnD+Hh4YwaNerUa+bNm0dCQgIhISFs2LDh1PrBgwcze/Zshg0bBsCcOXMYNGiQNK8IcYE+/jmTh9/fSbh/W1b+ciTDojqZXVKzkOFzXZx8pqK1e3fzEf78SRIjogJYfGc87b2d+zxWhs8VQrRKr3xz8NRNQgtvG9xic3uaRQJdCOFytNY89/leFm44yA0DQnj+1gF4tsDkE2ZzuEDXWjvNbbaOzqzmNCHMZLVqnvw0mbc2HWHGsHCevqkf7g40Znlzcqj/stq0aUN+fr4EkR1orcnPz6dNmzZmlyJEi7HUWvn9ql28tekIc0dH8dfJrSfMwcHO0MPCwsjIyCA3N9fsUlxCmzZtCAsLM7sMIVpElaWW3y7fwbrkbB4a34vfXNWz1f2171CB7unpSVRUlNllCCGcTHm1hV++s43v9ufx2PV9+cXlrTNHHCrQhRDiQhVX1vCLN7ay/WgBz97cn1uHhptdkmma1IaulEpQSu1VSh1QSs1vZPsLSqkddY99SikZzFsI0ezyS6uYuXgzOzMK+c+Mwa06zKEJZ+hKKXdgITAeyAC2KqVWa61TTu6jtX7QZv/fAIOaoVYhhDglu6iS25ZsJqOggkWz4hnbu7PZJZmuKWfow4ADWutDWutqYAVw4zn2nwEst0dxQgjRmKP55dzy3x85XlzFW78YJmFepymBHgqk2yxn1K07g1IqAogCvj7L9nlKqUSlVKL0ZBFCXIx9x0uY+uqPlFRaWDZ3OCO6B5hdksOwdz/06cAqrXVtYxu11ou01vFa6/igINca5UwI0fx2ZxQx7b+bAHhv3kj6h8lUjbaaEuiZgO2VhrC6dY2ZjjS3CCGawZZD+cxYvJl23h68/6uR9A72Nbskh9OUQN8KRCulopRSXhihvbrhTkqpPoA/sMm+JQohWrtv9uYw6/Wf6OLnzfu/GklEQDuzS3JI5w10rbUFuA9YD6QCK7XWyUqpp5RSk2x2nQ6s0HLfvhDCjtbsPsbctxPp2bk9K385kq4d2ppdksNq0o1FWus1wJoG6x5rsPyE/coSQghYmZjO/A92MbibP6/NHuoS8342J7lTVAjhkN744TBPfprC6OhA/nvHEHy8JK7ORz4hIYRD0Vrz0tcH+OcX+5gQ24V/zxiEt4drT0xhLxLoQgiHobXmb2v3sGjjIaYMCuXZqf3xaAUTU9iLBLoQwiHUWjV//iSJZVuOMmtkBE/cEItbKxrL3B4k0IUQpquptfLwyp2s3pnFr8f04PcTere6scztQQJdCGGqyppa7lu2nS9Tc/hjQh/uGdPD7JKclgS6EMI0ZVUW5r6dyI8H8/nLjbHcMTLS7JKcmgS6EMIUReU1zH7zJ3ZlFPHCtAFMHiTTJV4qCXQhRIvLLanijte2cCi3jJdvG8yE2GCzS3IJEuhCiBaVWVjB7Uu2kF1UyWuz4xkdLSOv2osEuhCixRzKLeX2JVsoqbLw7pxhDInoZHZJLkUCXQjRIlKPFXPHa1uwalg+dwRxoR3MLsnlSKALIZrdz0cLuPP1n/Dx8uDdOcPp2bm92SW5JAl0IUSz+vFgHnPeSiTI15t37x5OeCcfs0tyWRLoQohms+lgPr94cyvdOvnw7t3D6ezXxuySXJoEuhCiWfx0+MSpMF8+dwQB7b3NLsnlyTBmQgi7S0w7wew3fiKkYxuWzpEwbykS6EIIu9p+tIDZb2wl2K8Ny+eOIMhXwrylSKALIexmZ3ohd772EwHtvVg2d4S0mbcwCXQhhF0kZRZxx2tb6NjOk+VzRxDcQcK8pUmgCyEuWXJWEbct2YJvGyPMQzq2NbukVkkCXQhxSfZkF3P7ki2083Jn+dwRhPlLP3OzSKALIS7a/uMl3LZ4C94e7iybO4JuARLmZpJAF0JclAM5pcxYvAV3N8WyucOJDGxndkmtngS6EOKCHcotZebizQAsmzuC7kEyNosjkEAXQlyQtLwyZizeTK1Vs2yuDLTlSCTQhRBNln6inJmLN1NtsbJ07nB6dfE1uyRhQ8ZyEUI0SUZBOdMXbaasupZlc4fTJ9jP7JJEA3KGLoQ4r6zCCmYs3kxJZQ1L5wwnNkQmp3BETQp0pVSCUmqvUuqAUmr+Wfa5VSmVopRKVkots2+ZQgizZBdVMmPxZgrLanjn7uEy05ADO2+Ti1LKHVgIjAcygK1KqdVa6xSbfaKBR4BRWusCpVTn5ipYCNFycoqNMM8vrebtu4cxILyj2SWJc2jKGfow4IDW+pDWuhpYAdzYYJ+5wEKtdQGA1jrHvmUKIVpabkkVMxZv5nhxJW/eNZTB3fzNLkmcR1MCPRRIt1nOqFtnqxfQSyn1g1Jqs1IqobE3UkrNU0olKqUSc3NzL65iIUSzyyutYubizWQVVvLG7KHER3YyuyTRBPa6KOoBRANjgBnAYqXUGX+baa0Xaa3jtdbxQUFBdjq0EMKeTpRVc/uSLaQXlPP67KEM7x5gdkmiiZoS6JlAuM1yWN06WxnAaq11jdb6MLAPI+CFEE6ksNwI88N5ZSyZNZSRPSTMnUlTAn0rEK2UilJKeQHTgdUN9vkY4+wcpVQgRhPMITvWKYRoZkXlNdz+2hYO5JSyaFY8l0cHml2SuEDnDXSttQW4D1gPpAIrtdbJSqmnlFKT6nZbD+QrpVKADcDvtdb5zVW0EMK+iitrmPX6FvZml/DfO4ZwZS9pEnVGSmttyoHj4+N1YmKiKccWQtQrqaxh1us/kZRZxCu3DeHqvl3MLkmcg1Jqm9Y6vrFtcuu/EK1YWZWFu97Yyq6MIhbOHCxh7uTk1n8hWqnyagt3vbmVn9ML+ff0QSTEBZtdkrhEEuhCtEIV1bXc/WYiiWkneGHaQK7r39XskoQdSJOLEK1MZU0tc99OZPPhfJ6/dQCTBoSYXZKwEzlDF6IVqaypZd472/jhYB7/mDqAyYPCzC5J2JEEuhCtRJWlll8v3c7GfbksmNKPqUMkzF2NBLoQrUC1xcq9S3/m6z05/HVyP6YN7WZ2SaIZSKAL4eJqaq38Zvl2vkw9zlM3xjJzuIS5q5JAF8KFWWqtPLBiB+uTj/PY9X2ZNTLS7JJEM5JAF8JFWWqtPLhyJ//bfYw/XRfDLy6PMrsk0cyk26IQLiinpJLfv7+Lb/flMv/aPswZ3d3skkQLkEAXwsV8lXqcP6zaRWmVhadviuP2ERFmlyRaiAS6EC6iorqWv65J5Z3NR4jp6seK6QOJ7uJrdlmiBUmgC+ECkrOK+O2KHRzIKWXu6Ch+N6E33h7uZpclWpgEuhBOzGrVLPn+EP9Yv5dO7bx49+7hMjFFKyaBLoSTyi6q5OH3d/DDgXwmxHZhwZT++LfzMrssYSIJdCGc0Nrdx5j/4W6qLVb+fnM/bo0PRylldlnCZBLoQjiRsioLT36azMrEDPqHdeDFaQPpHtTe7LKEg5BAF8JJ7Egv5IEVP3PkRDn3ju3BA1f3wtNd7g0U9STQhXBwtVbNK98c4IUv9xPs14YVc0cwvHuA2WUJBySBLoQDyygo56H3dvJT2gluGBDC0zfF0aGtp9llCQclgS6Eg/pkRyZ/+igJDbwwbQA3DQyVC5/inCTQhXAwxZU1PPZxEh/vyGJIhD8vThtIeCcfs8sSTkACXQgHsjXtBA+s2EF2cSUPje/Fr8f0wEMufIomkkAXwgHU1Fr5z1f7eWnDAcL8fXj/VyMZ3M3f7LKEk5FAF8JkaXllPPDeDnakFzJ1SBhPTIqlvbf8aooLJz81QphEa8372zJ4YnUyHm6Kl2YO4vr+IWaXJZyYBLoQJigsr+bRj3azZnc2I7p34vlbBxLSsa3ZZQknJ4EuRAv78WAeD723k7zSKuZf24e5o7vj7ibdEcWla9Llc6VUglJqr1LqgFJqfiPbZyulcpVSO+oec+xfqhDOrdpi5W9rU7ltyRZ8vNz56Nej+NWVPSTMhd2c9wxdKeUOLATGAxnAVqXUaq11SoNd39Na39cMNQrh9A7klPLAez+TlFnMzOHd+NN1Mfh4yR/Iwr6a8hM1DDigtT4EoJRaAdwINAx0IUQDWmuWbjnK0/9Loa2nO4vuGMI1scFmlyVcVFMCPRRIt1nOAIY3st/NSqkrgH3Ag1rr9IY7KKXmAfMAunXrduHVCuFE8kur+OMHu/gyNYfR0YH885YBdPZrY3ZZwoXZ6xa0T4FIrXV/4AvgrcZ20lov0lrHa63jg4KC7HRoIRzPN3tzmPDid2zcl8dj1/flrbuGSZiLZteUM/RMINxmOaxu3Sla63ybxSXAs5demhDOp6i8hgXr9rD8p6P06tKed+4eRkxXP7PLEq1EUwJ9KxCtlIrCCPLpwEzbHZRSXbXWx+oWJwGpdq1SCAenteZ/u4/xxOoUTpRVMe+K7jw0vhdtPN3NLk20IucNdK21RSl1H7AecAde11onK6WeAhK11quB+5VSkwALcAKY3Yw1C+FQMgsr+PPHSXy9J4d+oR14866hxIV2MLss0QoprbUpB46Pj9eJiYmmHFsIe6i1at78MY1/fr4XreHha3ox+7JIGR1RNCul1DatdXxj26QjrBAXITmriEc+3M2ujCLG9A7i6ZviCPOXMcuFuSTQhbgAFdW1vPjlPpZ8fxh/H0/+M2MQ1/fvKjMJCYcggS5EE327L5c/fbyb9BMVzBgWzvyEGDr4yPyewnFIoAtxHnmlVTz9WQof78iie1A73ps3guHdA8wuS4gzSKALcRZaa1Zty+CZNamUVVm4f1w0vx7TQ7oiCoclgS5EIw7nlfHoh7vZdCif+Ah//jalH9FdfM0uS4hzkkAXwka1xcri7w7xr6/24+3hxjOT45gxtBtuMsStcAIS6ELU2XakgEc/3M3e4yVc168rj9/QV8ZfEU5FAl20eiWVNfxj/V7e2XyEYL82LJkVz9V9u5hdlhAXTAJdtGrrk7N5/JNkjpdUMvuySB6+pjftveXXQjgn+ckVrVJ2USWPr05iffJxYrr68eodQxgY3tHssoS4JBLoolWxWjVLtxzh7+v2UlNrZf61fbj78ig8ZfwV4QIk0EWrsTe7hEc+3MX2o4WMjg7k6ZviiAhoZ3ZZQtiNBLpweZU1tfzn6/3899tD+LX15IVpA7hpYKiMvyJcjgS6cGk/Hsjj0Y92k5Zfzs2Dw/i/62Lo1M7L7LKEaBYS6MIlFZRV88yaVFZtyyAiwIelc4Yzqmeg2WUJ0awk0IVL0VrzyY4snvosheKKGn49pgf3j4uW8VdEqyCBLlzGwdxSnlidzHf78xgY3pG/TeknEzSLVkUCXTi9ovIa/vXVft7elEYbT3eenBTL7SMicJfxV0QrI4EunJal1sryn47y/Bf7KKyoYfrQbjx8TS8C23ubXZoQppBAF07pu/25/OWzFPYdL2VE9048dn0sfUOkeUW0bhLowqkcyi3lr2tS+TI1h26dfHj19iFMiO0ifcqFQAJdOImiihr+89V+3tqUhreHO/Ov7cNdoyLx9nDR3itWK9SUQVUpVJdCVbHxvKqkbrmkwfNSqC6pf35ym7sndB0IoYMhdIjx3Lu92d+daCYS6MKhWWqtrNiazvNf7KOgvJpp8eE8dE0vOvs68DjlWkPBYSgvMIK4utQmZG1C1zaYT4WzTRijz38s5W4EtLcfeLU3nrfxA78QY111KWT9DCkf1+3vBoG9jXAPHWR87RwLHnKzlSuQQBcO64cDefzlsxT2ZJcwLKoTj13fl7jQDmaXdW5WK6x5GBJfP/s+bh7g7QtevnVh7Att/aFjN2PZy9dYd3Kb7X5edV9PPvdsC01pbirLg8ztkLUdMrfBvrWw411jm7s3BPerC/khxtl8px7gJgOWORuldRPOAppBfHy8TkxMNOXYwrEdzivjmf+l8mXqccL82/J/E2NIiAt2/HbyWgt8ci/sWgHDfgk9x9kEsE1Qe3g3LYSbk9ZQeNQI96ztdWG/w2jmAfDuACEDTw95vxBza3Z2lcWQvx/y9kNoPAT2vKi3UUpt01rHN7ZNztCFwyiurOGlrw/wxg+H8XJ34w8JvfnFqCjnuMvTUg0fzjWaNsb+H1zxe/ND+1yUAv8I4xE3xVhnrYXcvTYhvw1+/DdYLcZ2364QMri+PT5kELSVMeRPozUUZ0HePiO48/bVP0qO1e834W8XHejnIoEuTFdr1by3NZ1/fr6XE+XV3DIkjN9d09t55vOsqYT374R96+Cap+Gy35hd0cVxc4cufY3H4DuMdTUVkJ1khPvJoN/7v/rXBPSsC/m6M/ngfuDpJP9ul8JSDScO2gT2/vqv1aX1+3n7QWAv6D4WAqON50G9wT+yWcqSQBem+vFgHk99WtdOHtmJN6/vS78wB28nt1VdBitmwqFvYOJzMGyu2RXZl2dbCB9qPE6qKDCaZzK3GU01hzfC7pXGNjcP6BJ7esgH9Tb+s3BGFQVGSOfuPT24C9JA19bv5xdmBPag2+uDO7AXtO/Son+pNakNXSmVAPwLcAeWaK0XnGW/m4FVwFCt9TkbyKUNvXU7km+0k3+ecpzQjm15dGIME/s5QTu5rcpiWDYN0jfDpJdg0G1mV2Se4qz6gM/cZgR+VZGxzbMddIqyuahre7H35PUFm4u9p+1X13unOa87WK1QlG5zlr23/nlZbv1+7l7GXySnAru38TygZ4t2Bb2kNnSllDuwEBgPZABblVKrtdYpDfbzBX4LbLn0koWrKjnVTp6Gh7vi9xN6c/flTtJObqv8BCydagTXlMXQb6rZFZnLL8R4xNxgLFutRpPEyYAvSje6Y5blGl06T3bPPHkR9nzcPM/SA6hB8J/rPwuv9lCWc/qZdu4+yD8Alor6Y7X1N8K6V0L9mXZQL+gY4fB/aTSlyWUYcEBrfQhAKbUCuBFIabDfX4C/A7+3a4XCJdRaNe8npvPc53vJK61m6pAw/jDBidrJbZXlwds3GWdy096BPteZXZHjcXOrO5ONhgHTzr6ftba+//1Z++U37Mtft678hNFTx/bGqiZTRjfRwF7Q/crTm0l8Ahz7gvY5NCXQQ4F0m+UMYLjtDkqpwUC41vp/SqmzBrpSah4wD6Bbt24XXq1wSpsP5fPUpymkHCsmPsKf12cPpX+Yk/aOKD4Gb98IhUdgxnLoebXZFTk3N3do08F4XCrbu2vPdhOXTyfj7Dugh3F9wMVc8kVRpZQb8Dww+3z7aq0XAYvAaEO/1GMLx3Y0v5y/rkllXXI2oR3b8p8Zg7i+f1fnaie3VXgU3poEpTlw2yqIGm12RcKWm1t9Gzxdza7GFE0J9Ewg3GY5rG7dSb5AHPBN3S9qMLBaKTXpfBdGhWsqqaxh4YaDvP79YdzdFA+P78XcK7o7Xzu5rfyDxpl5ZTHM+uT0Xh9COIimBPpWIFopFYUR5NOBmSc3aq2LgFOTNSqlvgF+J2He+tRaNau2pfOP9fvIK61iyuBQ/pjQhy7O2E5uK2ePEea11XDnauMOSiEc0HkDXWttUUrdB6zH6Lb4utY6WSn1FJCotV7d3EUKx2a1ajYdyueva1JJzipmSIQ/r90Zz4BwJ20nt3VsF7xzkypUtG0AAA6jSURBVDEI1l1roHOM2RUJcVZNakPXWq8B1jRY99hZ9h1z6WUJR1dr1WxNO8Ha3cdYl5zN8eIqQjq04V/TBzJpQIjztpPbytgG7042urvNWt0st2oLYU9yp6hosppaK5sP5bM2KZvPk7PJK63G28ONMb2DuDauKxNig2nr5cTt5LaO/AhLb4V2AUaY+0eYXZEQ5yWBLs6pylLLjwfyWbP7GF+kHqewvAYfL3fG9unMxLiujOkdRDtvF/sxOvg1LJ8JHcKMNnMZZVA4CRf7TRT2UFlTy7f7clm7+xhfpeZQUmXB19uDq/t24dq4YK7oFeTcPVbOZe9aWDkLAqJh1sfQvrPZFQnRZBLoAoCyKgsb9uawNimbDXtyKK+upaOPJ9f2C+bauK5c1jPAdad7Oyn5I/hgjjFi4O0fGjehCOFEJNBbseLKGr5OzWHN7mN8uy+XKouVwPZe3DQolIlxXRnevROe7q1k1pqdK+DjeyBsGNy20j53LgrRwiTQW5nC8mo+TznOuqRsvt+fR3WtlWC/NswY1o2EuGCGRnbC3c0FeqhciMQ34LMHjTs/Z6wAr3ZmVyTERZFAbwXySqtYn5zNuqRsNh3Mx2LVhHZsy52XRZAQ15VB4R1xa20hftKml2H9IxB9Ddz6tkuO7yFaDwl0F3W8uJJ1Sdms2X2MrWknsGqIDPBh7hXdmRjXlbhQP9foK34pNj4HX/8FYibBza8hM98LZyeB7kIyCspZl5TN2qRsth0pACC6c3vuuyqaa+OC6RPse+EhXmuByiJj5paGj8pCm+dFENTHGEM6bCi4O/CPltbw9dPw3XPQ71a46RXHrleIJpKfYieXfqKcz3YdY23SMXZlGDPE9O3qx8Pje3Ftv2B6dvY1dqyphJLsxsP4rI+i+llnzsa7gzFRsFd7OPAl/PCiMUFAz/HQawL0HGcsOwqtYf2jsPllGDwLrn/R4SctEKKpJNCdkNVSw+adKXy1dQeZ6YfoRAlTO1p4KtpKT98a2ltL4Wgh7LUJZ9sZWRpS7kYot/U3Hu27GGfbJ5dPPtp0bLDc4fQz28oi46acfeth/+fGPJPKHbqNNMK9V4IxkYBZTT1WK/zvIdj2Bgz/FSQscNqJDIRoTJPmFG0OMqdoI7Q2zpxLso05GkuOGRMqlGRBSTaWwkyqCjJpW52PG438u7l7G32nTwvijo2Hse3D29f+wWatNaYe27fOCPjjScZ6/ygj2HtNgIhRLdduXWuB1ffBzuVw+YMw7nEJc+GUzjWnqAR6S7FUQ2l2fUAXHzMCu0FoU1N+5ku9O5JDJw5U+JJl9cejYwi9evaib+/eeHQMhXZBRjA7cg+NwnTYv94I90PfQm2VMe9jj7FGwEdfA+2DmufYlmr4cC6kfAxj/wRX/E7CXDgtCfTmpLXRpHHyjNo2oG2/lued+Vp3b/ANNsYK8e1a9zUYfLtS5RPM15nuvLazgsTMCny83Jk8KJQ7RkbQJ9iv5b9Pe6oug8Mb68/eS44BCkKH1J+9B/ezT+jWVML7s2HfWrjmGbjsvkt/TyFMJIFuL1pD7l44/K1xlpmTbJxVWyrP3NcnEPy6gm9I/deG4d3W/4zQSj9RzrtbjrByazoF5TX0CGrHrJGRTB4cil8bzxb6RluQ1pC9ywj2feuMZhowPq+T7e5RV4CXz4W/d3U5rJgJhzbAdf+EoXPsW7sQJpBAvxSF6fUBfnij0WwC0DECwuLBL9TmzLouvNsHX1DbsNWq+XZ/Lu9sOsKGvTm4KcX4mC7MGhnByB4Brau/eGmOcUF133rjAmt1KXi0gagr6wJ+gjEK4vlUFsOyaZC+GSa9BINua/7ahWgBEugXoiwf0jbWBfi3cOKQsb5dkHGmGHUldL8S/CMv+VCF5dW8n5jBu1uOcCS/nMD23swcFs6M4d3o2sGB28NbiqXKGJd833qjyaQgzVjfpR/0usY4ew8dcma3w4oCePdmOLYTpiyCuJtbvHQhmosE+rlUlcLRTXDoGyPAs5MAbVywixxVH+Cd+9rtQtrujCLe3pTG6p1ZVFmsDI30546RkSTEBuPl0UoGw7pQWkPe/vp296ObQNeCT4BxQbXXBOhxFdTWGFPG5e6FW96EPteZXbkQdiWBbstSbbTTngzwjESw1oC7F4QPrw/wkEHgbr8268qaWtbsPsbbm46wI72Qtp7uTB4cyh0jIojp6uQXOc1QUQAHvqrv815ZCG4eRvfM6lKYvhR6Xm12lULY3bkC3fVvLLJajT7QJ9vBj/wINWWAMmZvH3mvEeDhIy7uwtt5pJ8oZ9lPR3lvazonyqrpHtSOx2/oy81DwlzzImdLaesP/aYaj1oLZGw1zt6PJxn9zCMvN7tCIVqc6wW61ka798kAT/sOyvONbQHRMHAGdB9j/MI30y3pVqvmuwN5vLMpja/35ABwdUwXZo2MZFTPVnaRsyW4e0DESOMhRCvmGoFekm30QDl5IbMo3VjvG2K0r55sRmnmuSGLymt4f1s6724+Qlp+OYHtvfj1mJ7MHN6NkI5ykVMI0bycM9ArCuHID/UBnrvHWN+mozFJwajfGmfhAT1b5I7ApMwi3tl0hE92ZlJZY2VIhD8Pju9FQlyw60/bJoRwGM4X6D++BF/8GbQVPNoaf2YPmGGcgQf3b7GR86ostazdnc3bm9LYfrTuIuegUG4fEUFsiExfJoRoec4X6GHxMPp3RoCHDQUP7xY9vNaatUnZ/HVNKhkFFUQFtuPP1/dl6pAwOrSVi5xCCPM4X6B3G2E8TJCUWcRTn6bwU9oJenfx5Y3ZQ7myV1Drnb5NCOFQnC/QTZBTXMk/1u9l1fYM/H28eGZyHNPiw/Fwl5uAhBCOQwL9HCprannt+8O8vOEA1bVW5o7uzn1X9ZT+40IIh9SkQFdKJQD/AtyBJVrrBQ22/wq4F6gFSoF5WusUO9faYrTWrNlttJNnFlZwTd8uPDoxhsjAdmaXJoQQZ3XeQFdKuQMLgfFABrBVKbW6QWAv01q/Wrf/JOB5IKEZ6m12uzOKeOqzZLamFdAn2Jdlc4ZzWc9As8sSQojzasoZ+jDggNb6EIBSagVwI3Aq0LXWxTb7t4PG5kdzbMfr2sk/2J5BJx8v/jalH7fGh+MuFzyFEE6iKYEeCqTbLGcAwxvupJS6F3gI8AKuauyNlFLzgHkA3bp1u9Bam0VlTS1LvjvEy98cxFKrmXdFd+4dK+3kQgjnY7eLolrrhcBCpdRM4E/AnY3sswhYBMZoi/Y69sXQWvPZrmMsWLuHzMIKEmKDeWRiHyICpJ1cCOGcmhLomUC4zXJY3bqzWQG8cilFNbed6YX85bMUEo8UENPVj3/c0p/Lekg7uRDCuTUl0LcC0UqpKIwgnw7MtN1BKRWttd5ft3gdsB8HlF1UybPr9/Dh9kwC23uxYEo/bpF2ciGEizhvoGutLUqp+4D1GN0WX9daJyulngIStdargfuUUlcDNUABjTS3mKmyppZFGw/xyjcHqbVqfnVlD+4d2wNfaScXQriQJrWha63XAGsarHvM5vlv7VyXXWit+XTXMRasSSWrqJJr44J55NoYugXYfyILIYQwm8veKbojvZCnPk1m+9FC+nb14/lpAxnRPcDssoQQotm4XKBnF1Xy7Lo9fPhzJoHtvXn25v7cPCRM2smFEC7PZQK9otpoJ3/1W6Od/J4xPbh3bE/ae7vMtyiEEOfk9GmntWb1ziwWrN3DsaJKJvYz2snDO0k7uRCidXHqQP/5aAFPfZbCz0cLiQv148VpAxku7eRCiFbKKQM9q7CCZ9ft4eMdWQT5evPs1P5MHRwmE00IIVo1pwv0lVvTeWx1ElYN947twT1jpJ1cCCHACQM9IsCHcX26MP/aPtJOLoQQNpwu0Id3D5B2ciGEaIRMiimEEC5CAl0IIVyEBLoQQrgICXQhhHAREuhCCOEiJNCFEMJFSKALIYSLkEAXQggXobTW5hxYqVzgyEW+PBDIs2M5zk4+j9PJ51FPPovTucLnEaG1Dmpsg2mBfimUUola63iz63AU8nmcTj6PevJZnM7VPw9pchFCCBchgS6EEC7CWQN9kdkFOBj5PE4nn0c9+SxO59Kfh1O2oQshhDiTs56hCyGEaEACXQghXITTBbpSKkEptVcpdUApNd/sesyilApXSm1QSqUopZKVUr81uyZHoJRyV0r9rJT6zOxazKaU6qiUWqWU2qOUSlVKjTS7JrMopR6s+z1JUkotV0q1Mbum5uBUga6UcgcWAtcCfYEZSqm+5lZlGgvwsNa6LzACuLcVfxa2fgukml2Eg/gXsE5r3QcYQCv9XJRSocD9QLzWOg5wB6abW1XzcKpAB4YBB7TWh7TW1cAK4EaTazKF1vqY1np73fMSjF/WUHOrMpdSKgy4Dlhidi1mU0p1AK4AXgPQWldrrQvNrcpUHkBbpZQH4ANkmVxPs3C2QA8F0m2WM2jlIQaglIoEBgFbzK3EdC8CfwCsZhfiAKKAXOCNuiaoJUqpdmYXZQatdSbwHHAUOAYUaa0/N7eq5uFsgS4aUEq1Bz4AHtBaF5tdj1mUUtcDOVrrbWbX4iA8gMHAK1rrQUAZ0CqvOSml/DH+ko8CQoB2Sqnbza2qeThboGcC4TbLYXXrWiWllCdGmC/VWn9odj0mGwVMUkqlYTTFXaWUetfckkyVAWRorU/+1bYKI+Bbo6uBw1rrXK11DfAhcJnJNTULZwv0rUC0UipKKeWFcWFjtck1mUIppTDaR1O11s+bXY/ZtNaPaK3DtNaRGD8XX2utXfIsrCm01tlAulKqd92qcUCKiSWZ6SgwQinlU/d7Mw4XvUDsYXYBF0JrbVFK3Qesx7hS/brWOtnksswyCrgD2K2U2lG37lGt9RoTaxKO5TfA0rqTn0PAXSbXYwqt9Ral1CpgO0bvsJ9x0SEA5NZ/IYRwEc7W5CKEEOIsJNCFEMJFSKALIYSLkEAXQggXIYEuhBAuQgJdCCFchAS6EEK4iP8HBYFucnGCCiQAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"code","metadata":{"id":"wI4iirKXwOop","executionInfo":{"status":"ok","timestamp":1605375034210,"user_tz":420,"elapsed":1343,"user":{"displayName":"Buomsoo Kim","photoUrl":"","userId":"18268696804115368229"}},"outputId":"d3387e9a-6638-4c8f-8786-7f9b3384a4d1","colab":{"base_uri":"https://localhost:8080/"}},"source":["cnn_multichannel.load_weights('best_model.hdf5')\n","cnn_multichannel.compile(optimizer = 'Adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n","results = cnn_multichannel.evaluate(X_test, y_test)\n","print('Test accuracy: ', results[1])"],"execution_count":44,"outputs":[{"output_type":"stream","text":["67/67 [==============================] - 1s 9ms/step - loss: 1.3988 - accuracy: 0.3966\n","Test accuracy:  0.39662447571754456\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"O7xmyo_zwOos"},"source":["## Summary\n","- All four model variations show high level of training accuracy (almost 1.0), but suffer from low level of validation/test accuracy (below 0.4)\n","- Summary of evaluation results (test acc)\n","    - ```CNN-rand```: 0.4060\n","    - ```CNN-static```: 0.3309\n","    - ```CNN-non-static```: 0.4036\n","    - ```CNN-multichannel```: 0.3966"]},{"cell_type":"code","metadata":{"id":"4elJvLBMzjOr"},"source":[""],"execution_count":null,"outputs":[]}]}